mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-08 11:23:40 +01:00
f7c2be12898cd05a56e6df01231e109a9dc7314e
6370
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 DottaandPaperclip
|
f7c2be1289 |
fix(runtime): reserve worst-case PAX archive overhead
Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
0776f5b6f9 |
test(runtime): use the supported promise type library
Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
919f117092 |
fix(runtime): bound parallel offline harness backup transfers
Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
Devin Foley
|
69dea878c0 |
fix(sentry): prove a native capacity retry from its successor run (#19051)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server sends unexpected run failures to Sentry as run-failure events > - A native Codex run can fail because the model is at capacity. Paperclip then schedules a bounded automatic retry > - #18295 keeps that scheduled retry out of Sentry. It requires the committed finalization fields on the failed run row > - On staging, the failed run row can lose `finalizationReasonCode` after the decision is committed. The retry run exists, but the report still goes to Sentry > - This pull request uses the successor retry run as the proof, and stops reading the run's own finalization fields > - The benefit is that a capacity failure that has a scheduled retry does not make Sentry noise. Exhausted and held failures still report ## Linked Issues or Issue Description Refs #18295 **What happened?** A native Codex run failed with `native_provider_overloaded`. The status decision scheduled the capacity retry, and the retry run was created. The server still sent a run-failure event to Sentry. The server release contained #18295. The failed run row had `finalizationPhase: "committed"`, but it did not have `finalizationReasonCode`. The retry run was created about two seconds before the failed run's terminal write. **Expected behavior** When the committed capacity decision creates a retry run, the failure stays local. The retry owns recovery. A later failure of the retry reports its own outcome. **Steps to reproduce** 1. Make a native Codex run fail with a typed `codex`/`server_overloaded` diagnostic. 2. Let the status decision schedule the `native_provider_overloaded` retry. 3. Let a late terminal write replace the run's `resultJson` with a version that does not have `finalizationReasonCode`. 4. Report the run failure. Before this change, Sentry gets an event. **Additional context** Only the committed `schedule_retry` effect in `status-decision-committer.ts` creates a successor with `scheduledRetryReason = native_provider_overloaded` (through `materializeNativeProviderCapacityRetry`). Thus the successor row is a stronger proof than the run's own finalization fields. ## What Changed - `server/src/services/run-failure-report.ts`: `isScheduledNativeCapacityRetry` no longer requires `finalizationPhase === "committed"` or `finalizationReasonCode === "native_provider_overloaded"` on the failed run. It still requires a failed native run with `errorCode = native_provider_overloaded`, the typed `codex`/`server_overloaded` diagnostic, a native task, no signal, and a phase that is not setup. It still requires the successor run for the same task with `retryOfRunId = run.id` and `scheduledRetryReason = native_provider_overloaded`. The successor lookup and its index use do not change. - `server/src/services/__tests__/run-failure-report.test.ts`: two new cases. A scheduled retry stays local when the run row has no finalization fields, and when the row has an earlier finalization phase. The "finalization is not committed" must-report case changes to "finalization has no fields and no retry run exists". That case still reports. - `doc/observability.md`: describes the successor-run proof. ## Verification - `cd server && npx vitest run src/services/__tests__/run-failure-report.test.ts`: 284 tests pass. - Red check: with the `master` version of `run-failure-report.ts`, the two new cases fail. Each one captures one Sentry event. All other 282 tests pass. - `npx vitest run src/services/native-runtime/native-provider-failure.test.ts src/services/native-runtime/status-arbiter.test.ts`: 73 tests pass. - I could not load `native-provider-overload.integration.test.ts` in my local sandbox because of a missing environment setting. CI runs it. - Exhausted, held, other-lane, other-task, no-task, legacy, setup, signal and untyped-diagnostic failures still report. The existing must-report cases prove this. ## Risks Low risk. The change removes two field checks. The successor-run check stays. Only the committed capacity decision creates a successor with this retry reason, so a failure without a scheduled retry still reports. No migration. No change to the retry behavior. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) |
||
|
Nicky LeachandClaude Opus 5.5
|
0173110ee3 |
fix(native): point failed run attachments at the docs (#19052)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - When a later run cannot attach to a native session, the run fails, and a later run may start a new provider thread. > - The first failure is easy to miss. In 5 of 11 cases on a test deployment, the run ended with a different error before it reported the attach failure. > - People and agents who diagnose a failed run look at its log and events, so that is where an explanation is useful. > - This pull request adds a pointer there at the first failure, with the real cause and a link to the docs. > - The benefit is that a failed attach explains itself, and readers can find the open follow-up in #19047. ## Linked Issues or Issue Description Refs #19047. I searched open and closed pull requests and issues for `prior_attach_failed` and related terms, and found no duplicate. ## What Changed - **Log:** `executeHeartbeatRuntime` writes one stderr line to the run log when a same-run retry follows a failed `run.attach`. - **Events:** `heartbeat.ts` adds a `lifecycle` event with the same message and the code `native_run_attach_failed`, using the stored redacted failure. - **Helper:** `describeFailedAttachment` extracts the real cause and returns nothing for other failures. - **Docs:** "Session settlement after a run" in `doc/execution-semantics.md` now explains this and links #19047. - **Link target:** the pointer links to the repo docs and not the issue. Agents read these logs, and anyone can comment on a public issue. - **Unchanged:** error codes and retry behavior. ## Verification - `cd server && pnpm exec vitest run src/services/heartbeat/runtime-execution.test.ts src/services/native-runtime/remote-warm-session-recovery.test.ts`: 57 passed. - `npx tsc --noEmit -p .` in `server` reports no errors. - New tests cover the log line at the throw site, and the helper's output for attach failures and other failures. - **Not covered:** the new event in `heartbeat.ts` has no test of its own. It uses the tested helper. - **Not verified:** I have not confirmed that agents read this log when they diagnose a failure. - **Not run locally:** the full test suite. CI provides that coverage. ## Risks - **Failure text in the log:** when the failure has no structured message, the log line includes the whole failure string. It passes through the run log's identity redaction. The event uses the stored failure that the executor already redacted. - **Link stability:** the link depends on the docs heading. Renaming that heading would break the anchor but not the page link. - **Migrations:** this change has no migration. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be reworked. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), through Claude Code, with tool use and code execution. A human directed the design and approved the scope. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
Devin FoleyandClaude Opus 5.5
|
72d80944a9 |
fix(skills): keep the skill cache lock transaction active (#18996)
Replaces #17907. That branch conflicted with `master` after #17918 added
preparation timing to the same function. This branch applies the same
fix on current `master` and keeps that timing.
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Before a run starts, the server prepares the agent's skills while
holding a per-company skill cache lock.
> - The lock is a Postgres advisory transaction lock on a dedicated
connection. The preparation work runs on the ordinary pool, so the lock
transaction sits idle.
> - Hosted Postgres (Neon) sets `idle_in_transaction_session_timeout`.
If preparation runs longer than that timeout, Postgres ends the lock
session.
> - The lock is then released early, and the commit fails with `write
CONNECTION_CLOSED`. The run fails during setup and the preparation work
is lost.
> - #16340 and #16956 fixed the same problem for the workspace source
lock and the lifecycle lock with `withWorkspaceLockKeepalive`.
> - This pull request applies that keepalive to the skill cache lock.
> - With it, slow skill preparation keeps its lock and the run no longer
fails.
## Linked Issues or Issue Description
Refs #16340
Refs #16956
Supersedes #17907
**What happened?**
A hosted staging run failed during setup with `write CONNECTION_CLOSED
<neon-pooler-host>:5432`. The run trace shows `skills.prepare` running
for 322 seconds before it failed. That is longer than Neon's 5-minute
`idle_in_transaction_session_timeout`. The stack ends in
`postgres/src/connection.js` (`TLSSocket.closed`).
**Expected behavior**
Skill preparation holds the skill cache lock until it finishes, and no
other preparation for the same company can enter the locked section
during that time. The run then continues to execution.
**Steps to reproduce**
1. Set `idle_in_transaction_session_timeout` on the database role to
`1s`.
2. Call `withSkillVersionCacheLock` with an action that waits longer
than 1 second.
3. Without this change, a second `tryOnly` call for the same company
gets the lock while the first call still holds it. The first call then
fails with `write CONNECTION_CLOSED` instead of returning its own result
or error.
## What Changed
- After `withSkillVersionCacheLock` gets the lock and records the
`skills.cache.acquire` phase, it reads
`idle_in_transaction_session_timeout`. It then runs the action inside
the existing `withWorkspaceLockKeepalive` helper. The helper sends
`select 1` on the lock transaction every third of the timeout, and at
least every 30 seconds. When the timeout is 0 (disabled), the helper
does nothing.
- The `.catch` from `master` that ends the timing phase on failure is
kept.
- Added `skill-version-cache-lock.test.ts`. It runs against real
embedded Postgres with a 1-second idle timeout, following the same
pattern as `isolated-remote-workspace-source-lock.test.ts`.
- The edited file no longer has an extra blank line at the end. This is
whitespace only.
## Verification
All checks below ran on this branch's tree, which is `master` at
|
||
|
DottaandPaperclip
|
31266d7159 |
fix(logs): drain existing admissions before finalization (#19055)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Run logs retain execution output for operators. > - Object storage uses bounded admission and asynchronous uploads. > - A diagnostic can wait for capacity when its native execution ends. > - Finalization closed the writer before that existing admission could finish. > - This pull request joins those admissions before it closes the log. > - The benefit is complete accepted output with the existing storage bounds. ## Linked Issues or Issue Description Refs #18944 and #18327. **What happened?** A runner close diagnostic began object-log admission before its recorder retired. The call waited at the 8 MiB pending-byte bound. Completion then closed the writer. The waiting call failed with `Object run log writer is closed`. A valid final log hash covered the stored prefix, but the diagnostic was absent. **Expected behavior** Finalization must retain calls that started before closure. It must reject new calls after closure. Cancellation and storage failures must still settle without a success claim. **Steps to reproduce** 1. Use the actual recorder and segmented object store with a controlled conditional upload. 2. Fill the pending-byte bound. Start native warm-session retirement while the diagnostic waits for capacity. 3. Complete the run through the real completion handler. Release the upload. 4. Check the diagnostic bytes, final hash, source coverage, and warning rows. ## What Changed - Track active admission promises in each object-log writer. - Reject new producers when finalization starts. Join existing bounded admissions, then flush their tail before manifest closure. - Add actual completion coverage and store controls for failure, Stop, global capacity, and independent writers. - Clarify the finalization contract in the storage documentation. Keep feature maps unchanged for this routine fix. ## Verification - Before the fix, the actual completion probe lost a pre-retirement close diagnostic and emitted `object_log_persistence_failed`. Useful canonical output, source coverage, and the final prefix hash remained valid. - After the fix, the exact probe retains the diagnostic and checks the stored bytes and full hash. Ordinary completion and stalled local append controls also pass. - Focused store controls pass, including shared 64 MiB capacity, new-producer rejection, storage failure, Stop, and independent writer progress. - Production and test source at `bc03f8d53a721f01f80a12105a69d8583865e651`: 3 actual completion controls and 53 store, recorder, journal, and diagnostic tests pass. Run with `node node_modules/vitest/vitest.mjs run` from `server/`; select the four safety files plus the completion file with `-t "retired recorder"` in a separate invocation. - Initial treatment repository typechecks and build pass. Current map and module checks pass. A broader four-suite run had 99 passing assertions but failed its existing 30-second completion teardown hook. A later isolated completion run passed all 61 cases at the same limits. The original teardown cause remains unresolved; no timeout was changed. - Repository typechecks and build pass at that source commit. Current head `6652adc7f3379ae250f7b726dad6c58c8482410f` removes only the incidental map paragraph; production and test source are identical. All current-head checks are terminal: 54 successful check runs, one successful legacy status, and three skipped checks. Required `ci / verify` and `ci / e2e` pass. Greptile reviewed exact current head at 5/5. Both review threads are resolved. GitHub reports the PR clean and mergeable. Full repository tests stopped in chat shard 2: a GitHub recovery fair-batch test expected no provider requests but observed `GET /app/hook/config`. Its exact unchanged case passes in isolation. The original cause remains unresolved; this full-suite run is red. First-head remote server lane 2 failed a native-reassessment backlog test: its quiet check observed two active work items instead of zero. The exact unchanged case passes locally. That original CI cause remains unresolved; no job retry or timeout change was made. Other first-head CI lanes passed. The first native review rated the earlier source head 4/5 for the same removed map note. That is not a current-head assessment. Current GitHub rules require zero approving reviews and only the two CI checks above; a fresh native advisory re-review is optional and remains unavailable while its earlier run has an unresolved workspace/accounting obligation. That obligation is tracked separately and is not claimed recovered. - Removed the incidental map paragraph identified by Greptile and the native review. The follow-up contains no runtime or test changes. - This proves the local production-path mechanism. It does not attribute any specific hosted warning to this cause. Hosted replay and served revision checks remain separate acceptance work. ## Risks Finalization now waits for existing bounded capacity admissions. Storage cancellation and the existing operation deadlines still apply. Tests check that failed uploads and Stop settle, and that a separate writer can make progress. No capacity limit, timeout, or execution authority changes. ## Model Used OpenAI `gpt-6.1-sol`, through Codex, with reasoning and tool execution. Context window size is not exposed by the host. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (56 focused checks; the full-suite red remains documented above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandClaude Opus 5.5
|
86afc2048b |
fix(native): admit follow-ups after an owner finished in this process (#19071)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native Runner per-turn sessions in a remote sandbox keep one conversation across task turns. > - Before a follow-up starts a new Runner, it must prove that the previous owner's controller stopped. #18846 added a retirement path for this proof. > - The native controller runs inside the webserver process. Thus the owner's recorded controller PID is the PID of the webserver. > - When the same server process serves the follow-up, that PID is alive and has the same start time. The proof reads this as "controller still alive" and refuses. The follow-up then fails with `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate`. This continues until a deploy replaces the process. > - This pull request uses the controller's per-process boot identity to recognize that this process recorded the owner. The process then uses its own session registry to prove that the owner's controller stopped. > - The benefit is that follow-ups on a long-lived server no longer fail before provider work. All other ownership fences stay as they are. ## Linked Issues or Issue Description Refs #18846 (retirement of recovered per-turn owners). Refs #19061 (names the refused retirement proof). The two changes are compatible. Whichever merges second needs a small rebase in `retireFinishedPerTurnRunnerdOwner`. **What happened?** On a test deployment, 17 runs over 14 tasks failed before provider work with `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate`. Each run had a per-turn lifecycle in a Daytona sandbox and got a new sandbox. In each inspected case, the previous run on the task succeeded and committed its result. Its run profile had no `nativeControllerRecovery` pointer. Without that pointer, `evaluateNativeControllerTakeover` is not lease-fenced, so it uses process liveness. The recorded `controllerPid` is `process.pid` of the webserver. When the follow-up is served in the same process, `process.kill(pid, 0)` succeeds and the start time matches. The result is `controller_still_alive`, and retirement refuses. **Expected behavior** A committed owner that this server process finished, released, and no longer runs is proven stopped. The follow-up then starts a new Runner, as #18846 intended. **Steps to reproduce** 1. Seed a per-turn sandbox owner that committed its result and released its lease. Record this process's controller identity (`currentNativeControllerIdentity()`) as the owner. 2. Keep the owner's file-backed journal in the local session root, and remove the database recovery pointer. 3. Admit a follow-up in a new sandbox. 4. Before this change, admission fails with `prior_owner_terminal_state_indeterminate`. **Paperclip version or commit** Observed on `e107a75755`. The same code is on `master`. ## What Changed - `native-restart-recovery.ts`: add `isCurrentNativeControllerProcess(owner)`. It is true only when the owner's `controllerBootId` is this process's per-start boot UUID and `controllerPid === process.pid`. A recycled PID or another container never matches. - `retireFinishedPerTurnRunnerdOwner`: when the owner was finished in this process and no `activeNativeSessions` or `nativeSessionStartups` entry exists for its run, treat its controller as stopped. Otherwise, use `evaluateNativeControllerTakeover` as before. The same in-process check runs again just before the archive. - All other retirement proofs are unchanged: committed result, released lease, no recovery state, settled journal, answered tool inputs, quarantine copies, and new or replacement sandbox. - `native-recovered-owner-followup.test.ts`: the seed helper accepts the owner's controller identity. New cases cover same-process admission (with and without a recovery pointer), an exited legacy controller, another live controller, and the same PID with a different boot identity. - `doc/execution-semantics.md`: describe the same-process stopped-controller proof. ## Verification - **Failure before the change:** with only the new tests applied, `cd server && npx vitest run src/services/native-runtime/native-recovered-owner-followup.test.ts` fails 1 case: "admits a follow-up in the process that finished the owner: local journal only". It expected admission and received `runner_state_identity_mismatch: prior_owner_terminal_state_indeterminate`, the exact error from the deployment. The safety cases already passed on the old code. - **After the change:** the same file passes 27 of 27. - Adjacent suites: `native-recovered-owner-followup`, `remote-warm-session-recovery`, `native-runner-restart-recovery.integration`, `native-restart-recovery`, `native-session-resume`: 210 passed, 20 skipped. - `native-session-executor.test.ts`: 1128 passed. - `tsc --noEmit -p server/tsconfig.json`: no errors. - **Not verified live:** this change has not run on a deployment. Run data came from read-only API calls only. No recovery was done on the deployment. ## Risks - **Second executor.** The new proof applies only when this exact process recorded the owner, and this process has no session or startup for that run. The database proofs still require a committed result and a released lease. Another live controller, or the same PID with a different boot identity, still holds the owner. Tests cover both cases. - **Scope.** The change applies only to the per-turn retirement path. The warm remote recovery path (`inspectRemoteWarmSessionRecovery`) is not changed. Warm handoff in the same process uses the in-memory idle owner instead. - **Hypothesis limit.** The deployment API does not expose finalization rows. Thus I inferred that the same process served each follow-up from the evidence: no recovery pointer, a local journal present, and no other failing proof. The local reproduction gives the exact error. #19061 can confirm the refused fence on the deployment. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic. Agentic coding in Claude Code with tool use (shell, browser for read-only staging API reads, code execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
c19c4348c5 |
fix(server): record inspector provenance for dynamic import callback errors (#17844)
<!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/49/changes/paperclip/49 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server sends errors to Sentry when an operator sets a backend DSN. An unhandled rejection is captured in `strict` mode, so the server process exits after the capture. > - A hosted server reported `TypeError [ERR_VM_DYNAMIC_IMPORT_CALLBACK_MISSING]: A dynamic import callback was not specified.` two times, eight days apart, on two unrelated releases. Each event stopped the process. > - The stack has no first-party frame. An anonymous `setTimeout` callback at `<anonymous>:1:N` calls `import()`. The column changed between the two events, so the source text changed. > - In local reproduction, only two paths make that frame: `vm.compileFunction` without a filename, and an expression that a client evaluates through the Node inspector. No shipped server code or production dependency uses either path. The current events give no data about the source. > - This pull request adds bounded provenance to that one error: the inspector state when the event is reported (never its URL), allow-listed Node flag names, the process ids, the main-thread flag and the uptime. > - The benefit is that the next event gives evidence about the source of the anonymous code. A `closed` state rules out an expression from a still-open inspector session. An `open` state needs other evidence before a debugger is blamed. ## Linked Issues or Issue Description No public issue exists. This section follows the bug report template. **What happened?** A hosted server process stopped two times with an unhandled rejection: `TypeError [ERR_VM_DYNAMIC_IMPORT_CALLBACK_MISSING]: A dynamic import callback was not specified.` The stack is `importModuleDynamicallyCallback (node:internal/modules/esm/utils:280)` ← `Timeout._onTimeout (<anonymous>:1:20)` (first event) or `(<anonymous>:1:41)` (second event) ← `listOnTimeout`. The events came 3 minutes and 17 minutes after process start, on two releases that changed only skill documents and UI files. The Sentry event has no data that shows where the anonymous code came from. **Expected behavior** The error report gives enough data to investigate the source of the anonymous code: whether an inspector session was open in the process, and which inspector-related Node flags were set. **Steps to reproduce** 1. `node -e "require('node:vm').compileFunction(\"setTimeout(() => import('node:fs'), 1)\", [])()"` gives the same error and the same `Timeout._onTimeout (<anonymous>:1:N)` frame. 2. An expression such as `setTimeout(() => import('node:fs'), 1)`, evaluated through the inspector (`Runtime.evaluate`), gives the same error and frame. 3. `vm.Script` (`evalmachine.<anonymous>`), `eval` or `new Function` in a `vm` context (`eval at …`), `new Function`, indirect `eval`, `node -e`, `node -p` and stdin give a different frame or import successfully. **Paperclip version or commit** `master` at `0cb3fc7db591ba23e21fbbc70042bc44ec7d6f20`. The hosted events came from builds `f96a1e64` and `3e7c3ee9`, on Node v24.22.0 and v24.22.1. **Relevant subsystem** Server Sentry error monitoring (`server/src/sentry.ts`). ## What Changed - New `server/src/sentry-dynamic-import-provenance.ts`: `addDynamicImportProvenance(event, hint)` acts only when `hint.originalException.code` is `ERR_VM_DYNAMIC_IMPORT_CALLBACK_MISSING`. It adds: - the tag `dynamic_import_inspector` with the value `open`, `closed` or `unavailable` - the context `dynamic_import_provenance` with `inspector`, `nodeFlags`, `mainThread`, `pid`, `ppid` and `uptimeSeconds` - `nodeFlags` holds only the names of `--inspect`, `--inspect-brk`, `--inspect-wait`, `--inspect-port`, `--debug-port`, `--disable-sigusr1` and `--experimental-vm-modules` from `execArgv` and `NODE_OPTIONS`. Flag values and all other flags are dropped. - The hook never sends the inspector URL, because the URL contains a session secret. It loads `node:inspector` lazily, so a Node build without the inspector still starts and reports `unavailable`. The hook never throws. Every other event is unchanged. - The module comment and the doc say that the state is evidence at report time, not proof of origin. - `server/src/sentry.ts`: `beforeSend` is now `enrichServerEvent`. It runs the existing run-link enrichment, then the new hook. - `doc/observability.md`: the default capture set names the new tag and context, and says what is never sent. - `.github/workflows/sentry-contract.yml`: the Real Sentry SDK isolation job also runs the new test file. - Tests in `server/src/__tests__/sentry-dynamic-import-provenance.test.ts`: - the real Node error from `vm.compileFunction` - a real open inspector session: the URL and its secret are absent from the event. The test closes only the session it opened, and it skips when a developer already has an inspector open. - flag filtering with values and other flags present in `NODE_OPTIONS` - other codes and other hint shapes stay unchanged - a failed state read keeps the original event - the real `@sentry/node@10.71.0` SDK sends the original rejection to the server `beforeSend`, and enriches only the matching event. With `PAPERCLIP_REQUIRE_SENTRY_TEST_SDK=1`, a missing SDK fails the file. ## Verification - With the audited SDK on `NODE_PATH` and `PAPERCLIP_REQUIRE_SENTRY_TEST_SDK=1`, as in the contract job: `run-failure-sentry-real-sdk.test.ts` and the new test file pass (17 tests). - Without the SDK: the new test file, `sentry.test.ts` and `sentry-links.test.ts` pass (52 pass, 6 skipped). - With `PAPERCLIP_REQUIRE_SENTRY_TEST_SDK=1` and no SDK: the new test file fails at once, as intended. - Red: with `beforeSend` set back to `addSentryRunLinks`, the real-SDK test fails (`expected [] to have a length of 1`). Green with the change. - Hosted CI on this head: typecheck, build, all test partitions and Real Sentry SDK isolation pass. - Not run locally: the full Vitest suite and browser suites. A live hosted check is not possible before deployment. ## Risks - Low risk. The hook acts only on one error code. All other events keep their current content and grouping. - The process still exits after the capture, as before. This change adds diagnostics. It does not fix the source of the anonymous code. - `pid`, `ppid` and uptime describe the server process only. They hold no tenant, user or request data. - If a later Node version changes the error code, the hook stops acting and the events are the same as today. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through the Claude Agent SDK, with tool use (shell, file edit, code execution, GitHub and Sentry connectors). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
a3313f6944 |
fix(ui): complete read recovery and enforce query error rendering (#18822)
## Thinking Path > - Paperclip lets people manage agent work through the board. > - The board reads task, agent, settings, plugin, and run data from the server. > - A deploy can interrupt those reads after content is already visible. > - Local error branches can replace that content with an error or show raw codes. > - This pull request moves the remaining reads to the shared recovery behavior. > - The query-error guard prevents new local error patterns from returning. ## Linked Issues or Issue Description **What happened?** Loaded lists, settings, task panels, and manual reads could clear their content or show errors during a server restart. Some reads also disabled the shared retry policy. **Expected behavior** Keep loaded content during a temporary failure. Show the shared connection banner. Retry reads after recovery. Replace content after a real access denial or missing resource, with readable copy. **Steps to reproduce** Open a task, settings page, or run log. Return 503 with `tenant_app_unavailable` from the API. Restore the API and check that content refreshes. Repeat with 403 and 404 to check the failure state. Related merged work: Refs #15754, Refs #15772, Refs #17960. This change covers remaining reads and enforcement. Browser persistence and the outage browser suite follow separately. ## What Changed - Repair an existing PostgreSQL deadline fixture at UTC boundaries. A daily cap defers a heartbeat until both its original interval and the next UTC day allow it. Keep the counted run in its seeded day and cover noon, just before midnight and the first second after midnight with explicit clocks and real PostgreSQL. Production scheduling is unchanged. - Integrate the saved list, settings, shell, task, and manual-read recovery changes against current master. Preserve the current task, Inbox, and run-log structure. - Add `useResilientFetch` for component-local reads. Preserve loaded data, cancel stale requests, and retry through the shared connectivity policy. Keep browser and pairing capabilities in memory. - Retain run log chunks across stream changes and catch up after recovery without duplicate chunks or stalled requests. - Keep genuine authorization and missing-resource failures visible. Translate known single-word codes such as `forbidden`. Show known skills and disconnected-app failures before sibling loading states. Keep Retry bound to its failed request. - Make the query-error guard enforce by default and run it in the CI policy job. Document the narrow exceptions and the read recovery contract. - Invalidate pending OAuth discovery when the operator leaves setup or its context changes. Keep reserved-popup cleanup bound to the current attempt. ## Verification - Current read head `09d1db29dea` is merge-ready: all 59 check runs are completed (54 successful, three skipped, two neutral), the legacy Snyk status is successful, and both required `ci / verify` and `ci / e2e` gates pass. Greptile Apex reviewed this exact head at 5/5 with no unresolved threads; the branch has no merge conflicts. The current CI run is `41748339602`. - Current read head `09d1db29dea` includes master `683fa4725e9`. The latest change touches only the existing server deadline test fixture. Application source remains the previously verified `86376fa751e` revision: all 60 affected skills/disconnected-app tests and both application UI typechecks passed, along with token gates and the enforcing query-error guard. Those UI suites were not repeated for this test-only repair. - The fixture repair passes all seven deadline integration cases and the complete server typecheck, including the server test layer. The old assertion deterministically fails at 23:59:06.888 UTC; its one-second-old run also fails the first-second-of-day probe. The corrected fixture passes noon and both midnight probes. The permanent three-case coverage uses seeded 2099 clocks. - Twelve new regressions fail against the previous implementation. Both skills variants and disconnected-app pages show either request's 403 or 429 while the sibling pauses or remains fetching. Permission failures stay visible. Retry stays usable for its failed request and recovers independently. An initial test-fixture query-key typo was corrected before the passing run. The finding is replied to and resolved. Greptile reviewed prior application revision `86376fa751e` at 5/5 with no new actionable findings or unresolved threads. The current test-only head has now received its own Apex 5/5 review with no unresolved threads, and its CI is successful as recorded above. - The preceding OAuth fix passed all 237 setup tests on both application branches. Its ten regressions prove that Back, Cancel, unmount, or context changes retire initial discovery without creating a connection, caching old identities, navigating an abandoned popup, or closing a newer attempt. - Earlier master integration `e992f770339` passed all 309 tests across eight affected Slack, recovery, retry, submission and task files, plus UI typecheck, token gates, enforcing query-error scan and seven guard tests. The integration preserves Slack workspace reconnect and shared recovery, readable execution errors, source-inspection company scoping, and current submission behavior. Newly inherited retained-file recovery controls keep cached data through transient failures and hide it after definitive denial. - The preceding cache head `97c0bc3966f` passed 1,038 focused cache/write/editor/account/task/API tests. All 22 built-UI outage/write/loading cases passed in 4.7 minutes on exact browser head `b7deaaee5da`, with strict checking of all three specs. The latest skills/app rendering fix is covered by its affected component suites; those broader cases were not repeated for this separate read path. - Historical broader UI, CLI and build results and unresolved aggregate server/DB/adapter failures remain in the PR discussion. Interrupted CI runs were audited and recorded separately; they are not counted as passes. Controlled local outages do not prove hosted authentication or a real Cloud restart/deploy. Real hosted acceptance remains required after merge and deployment. Commands for the latest fix: ```sh pnpm --filter @paperclipai/ui exec vitest run src/pages/CompanySkills.test.tsx src/pages/apps/AppNotConnected.test.tsx pnpm --filter @paperclipai/ui typecheck pnpm check:token-gates node scripts/check-query-error-rendering.mjs --enforce --summary ``` ## Risks This migration touches many read surfaces. The main risks are initial loading, request cancellation, retained authorization state, and run-log paging. The tests cover those states. There are no server, schema, dependency, or API contract changes. The follow-up browser suite and hosted deploy acceptance are required before release acceptance. Recovered commits retain their original authors. If squash-merging, preserve `Co-Authored-By: Bender (Fable) <bender@agents.paperclip.ing>` and the Paperclip trailer. ## Model Used OpenAI `gpt-6.1-sol` through Codex, with xhigh reasoning effort, tool use, and code execution. The context window size is not exposed. Saved source commits keep their original authorship. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR following the bug report template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal ticket id - [x] The affected local tests pass; broader limitations are recorded above - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com> Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Bender (Fable) <bender@agents.paperclip.ing> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
52b4b3ccd7 |
test(evals): pin native credential delivery and worker receipt fixes (#19042)
## Thinking Path > - Paperclip must deliver agent work through ordinary human controls. > - The latest three-case run retained three failed originals. > - The personal value was saved, but the native OpenCode child did not receive the configured task environment. > - A worker proof also relied on legacy process identity fields outside that environment contract. > - Timeline and credential checks had independent review defects that could reject truthful outcomes or miss unsafe instructions. > - This PR pins the reviewed fixes together and keeps all prior grades unchanged. > - A new AWS measurement must still prove the real outcome. ## Linked Issues or Issue Description Refs #18963, #18967, #18968, #18971, #18972, #18977, #19027, #19031. Follows merged #18995. This is the next infrastructure pin, not another core implementation PR. The latest launch 2657 retained FAILED originals for timeline, approved-tool replay and personal secret. The personal journey reached secure UI save and a worker wake, but the worker reported no credential and made no endpoint call. These are failed attempts, not new passes. Six of the selected ten cases have independent passes across prior attempts. The uniform final ten and the full catalog remain unqualified. No old result was regraded. ## What Changed - Pin immutable candidate `c6cc59282a45ed3c79880a220f7f60aa3ac1d6a4` on `codex/chat-credential-destination-contract-candidate-20261104`. - Retain the frozen candidate baseline. Add the accepted failed-action completion feedback hint, canonical configured environment delivery to OpenCode, actual native worker output receipt checks, timeline attribution controls, and credential clause safety checks. The feedback change leaves the existing acceptance gate in place. - Declare exactly seven oracle-version changes: timeline v5 to v6; personal v5 to v6; company v4 to v5; secure API key, least-scope delivery, secret proposal and vault reference each external v1 to v2. The four external cases have explicit overrides; the shared default is unchanged. - Sync the manifest rows, explicit changed-field records, descriptor snapshot and expected target. Preserve all 244 identities, original queries, human/native counts, clocks, model, profile, launcher bytes, privacy limits and the existing 196/244 qualification gate. - Include the reviewed retry cancellation readiness fixture repair from `815d47e4b5d7756367d2c5e2da656b640273eac2` in this infrastructure PR only. It awaits the first scheduled batch before polling its state. Its production behavior and assertions are unchanged. This test repair is absent from the model candidate. - Keep the core PRs, including #19027 and #19031, unmerged. A normal merge of this pin does not merge those PRs. ## Verification - Pin-local checks: all 189 existing manifest, TestOps API, adapter, reconciliation and streaming controls passed. The reviewed 815 fixture repair passed its owning real-PG control (one selected test). A temporary delayed-first-batch control failed with the old observer and passed with the new observer. Batch limits, assertions, drain wait and overall budget are unchanged. The original CI cause remains unknown. - Final candidate: 89 credential/receipt consumer controls passed. The exact Runner TypeScript build, 68 driver controls and one owning failed-action PG control passed before the later helper-only changes. Their runtime inputs remain byte-identical; those results are reused explicitly, not claimed as new executions on every source commit. - Real pinned OpenCode 1.18.34 executed a canned local-provider bash call. Synthetic `PUBLISHING_API_KEY` reached its child in a 2.89-second passing control with zero paid provider calls. This proves server-to-bash inheritance; driver spawn controls cover the preceding projection boundary. Actual AWS secure-entry, worker delivery and privacy acceptance remain pending. - **The latest local candidate declaration loader is NOT GREEN.** It exceeded its unchanged 30-second limit in both a combined run and an isolated run. Five other declaration/contract checks passed. Both failures remain recorded. No timeout was widened and no further retry occurred. Descriptor and definition modules match the earlier passing 5aa source; only the transitive credential helper changed. Prior 242-declaration evidence supports unchanged definitions only, not a claim that these failed executions passed. - The snapshot matches the candidate and exactly seven oracle-version leaves differ from source 249. All 244 identities, queries, run/human counts, clocks, model, profile, launcher and privacy limits remain unchanged. The SDK guard reports 506 unchanged inputs, unchanged runtime pins and registry `efa70621e9fb42aa185f7ee616a1a726aa3eccb367e67e28817cedf26ef2ade8`. The driver is outside those SDK roots. No SDK renewal occurred. Reporting-lock and whitespace checks passed. - Genuine exact-head reviews: source 19e3 is 5/5 with no unresolved threads; current pin 988 is 5/5 with no unresolved threads. The prior pin 4/5 and resolved P1 remain historical. Current exact-head normal CI 41917215227 completed successfully: 48 successful jobs and one selector skip, including final verify job 126888275415. Windows 41917215440 succeeded. The full PR rollup is 56 successes and three intentional skips, with no pending or failed checks. Older green heads do not qualify this head. Paperclip has only an earlier-head 5/5 assessment, which is stale for this head and receives no current assessment credit. - Six of ten cases have independent passes across prior attempts. Launch 2657 retains three FAILED originals; no grade was changed. The next measurement is the exact uniform ten, cap eight and zero automatic retries. No uniform-ten or full-catalog acceptance is claimed. - The local-tests checkbox below covers only the 189 pin controls and owning fixture control. It does not claim all candidate or repository tests are green. Full local repository checks were not repeated for this infrastructure diff. Core PRs remain unmerged. No model call, eval dispatch, private archive read, fleet change or merge occurred for this update. ## Risks Environment delivery crosses a real provider process boundary, so secret isolation remains part of the next actual AWS acceptance. A worker receipt must match an actual successful native run and key-authenticated endpoint call; a copied boolean does not qualify. Oracle safety and attribution changes can alter acceptance, so balanced negatives remain required. No current uniform-ten pass is claimed. Normal CI and genuine exact-head review must finish before protected infrastructure merge. Root owns the next TestOps Submit, with the original model, limits, zero automatic retries and maximum eight concurrent case chains. ## Model Used OpenAI `gpt-6.1-sol` through Codex. Capabilities used: source analysis, mechanical integration, local code execution and PR preparation. The host does not expose the context-window limit. The evaluated model remains OpenCode/OpenRouter `deepseek/deepseek-v4.1-flash-1017`. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>canary/v2026.1104.0-canary.14 |
||
|
Dotta
|
c6cb7e2a2d |
fix(ui): accept file drops across task and new-task views (#18849)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - People add images and files through task message composers. > - A file drop requires a small target inside the composer. > - A screen-wide target makes this action easier. > - Task pages and new-task dialogs use the same composer. > - This pull request adds one shared file-drop target. > - The open dialog receives the drop when it covers a task page. ## Linked Issues or Issue Description Refs #13206. This improves the task composer drop area. Related PRs: #18689 adds image previews. #15964 adds direct uploads in the Artifacts panel. **What existing behavior does this improve?** Image and file drops in the task composer and new-task dialog. **Current behavior** People must drop a file into the composer area. **Proposed behavior** People can drop a file anywhere on the screen when an available composer is present. A full-screen overlay shows the destination. **Reason and benefit** People can add an image without aiming at the text field. **Subsystem affected** React board UI. **Breaking changes** The available composer captures file drops across the screen. Text and link drags retain their current behavior. Upload rules and APIs stay the same. **Additional context** ROADMAP.md lists task discussion and attachments as existing features. This change improves their input area. ## What Changed - Add a shared screen-wide drop target to the task composer. - Give an open task dialog exclusive ownership of file drops. Keep file drops available while a model or assignee picker is open. - Give explicit editor, task attachment, and Artifacts drop areas priority over the screen-wide composer target. Share one marker for these areas. - Keep image and file uploads in description, question, and protocol editors. Read-only editors and editors that delegate uploads to a parent keep the composer fallback. - Prevent browser navigation from file drops while the active composer is read-only. - Use the existing file staging and upload callbacks. - Disable drops during pause, queued message edits, uncertain sends, submission, and read-only states. - Exclude hidden and inert composers. - Clear the overlay after drop, drag exit, cancellation, window blur, and unmount. - Add component tests and two Storybook drag-over examples. ## Verification - 316 tests passed across the drop target, real MarkdownEditor, task composer, new-task dialog, and Artifacts tab. Tests cover image and text-file drops in a real editor, overlay reset, parent and read-only editors, and task attachments. - Two new cases failed before the editor and attachment fixes. Both passed after the fixes. - An earlier expanded run had three new-task upload assertion failures. The five affected cases passed on the unchanged implementation in isolation. The expanded run then passed all 316 tests. The cause of the earlier failures remains unverified. - UI typecheck and UI production build passed. The first typecheck attempt lacked plugin SDK declarations. Building the SDK restored these declarations; typecheck then passed. - Design-token gates, Storybook catalog generation, and whitespace checks passed. - Chromium checks from the design review confirmed one image per screen drop in each view. The editor stayed empty and the overlay cleared. These checks used local Storybook fixtures. - The requested screenshots were accepted before this PR was opened. - Full repository typecheck and build were attempted. Both stopped in the native Runner package because this environment has no cargo executable. - The full repository test run was attempted. It was stopped during the first chat integration shard before a full-suite verdict. The focused UI suite passed. - The first CI attempt did not complete. Typecheck and build logs report runner shutdown signals. Most browser shards were cancelled; their logs could not be retrieved. No complete CI verdict is available. The new commit starts a fresh CI run. - Review findings about Artifacts drops, portaled pickers, read-only drop navigation, and editor drop ownership are fixed. Current-head CI and review are pending. - The previous CI run passed build, typecheck, general tests, and six browser shards. Browser shards 5/8 and 8/8 failed. Their logs returned BlobNotFound; the causes remain unverified. The new commit starts a fresh CI run. Manual check: open a task, drag a PNG over the page header, and drop it. Confirm one image preview. Open a new-task dialog over the task and repeat. Confirm the file appears only in the new task. Cancel the drag and confirm the overlay clears. Open the description editor. Drop an image and then a text file into it. Confirm each upload stays with the description and does not enter the chat draft. ## Risks - Screen-wide file capture changes which surface receives file drops. - Dialog ownership and disabled-state tests limit unintended uploads. - No server, API, database, or migration changes are required. - Full repository acceptance still requires the native Rust toolchain and CI results. ## Model Used OpenAI GPT-6 through Codex. The exact runtime snapshot ID and context-window size were not exposed. Capabilities used: repository inspection, code editing, shell commands, and browser automation. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge |
||
|
DottaandPaperclip
|
61b2455319 |
fix(runtime): preserve failed sources and durable checkpoint authority (#19038)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A volume-free web server must preserve retained native session data. > - An old failed follow-up can keep its own snapshot while its controller file still belongs to the preceding successful turn. > - The Runner correctly refuses to import that file as the failed turn's execution authority. > - That refusal also blocks migration when no warm supervisor is present. > - This pull request preserves the file after it proves the older run's lineage, and uses existing typed database authority when present. > - Failed history and files survive, while normal execution authority stays unchanged. ## Linked Issues or Issue Description Refs #19024, which preserves the related older-supervisor case. The duplicate search found no PR for this older-controller case. **What happened?** Offline native-controller migration stops with `native_runner_checkpoint_binding_mismatch` for a failed retained follow-up. The local controller still contains the preceding successful run. A per-turn source can have no supervisor file, so the older-supervisor archive path does not apply. Read-only checks matched the database snapshots and controller-file hashes between the original source and the isolated rehearsal copy. **Expected behavior** Preserve the available file as evidence through the existing encrypted archive. Require a valid older controller, a completed successful prior run, and the same company, agent, task, native session, Runner and provider thread. Keep successful sources and active owners outside this archival fallback. When a valid typed database checkpoint already exists, derive only its missing recovery metadata and ignore obsolete local projections. **Steps to reproduce** Create a successful native turn and a later failed follow-up in the same session. Retain the successful controller file. Keep the failed snapshot and admitted input in the database. Leave the per-turn source without a supervisor. Run offline migration. The new positive regression fails with the binding mismatch before this change. ## What Changed - Share the scoped older-successful-run proof with the existing supervisor preservation path. - Validate the older controller through the Runner parser and artifact checks before archive publication. - Archive this failed legacy source without changing its snapshot, status, ownership, queued work or session pointer. - Preserve the normal refusal for foreign, changed, active, successful, invalid and already-durable cases. - Permit evidence archival for a stale observed coordinator only with exact host-owned stop receipts for every remote lease, no current task ownership, no result and no retry deadline. Recheck the proof before publication. - Use execution start for lifecycle ordering. A queued follow-up can predate its prior turn completion, and a remote stop can predate a later recovery timeout. - Resolve destruction receipts through the existing reusable-lease handoff verifier. Reject stopped tips, broken chains and competing tips. - Prefer a valid typed database checkpoint over a stale or absent local warm supervisor. Add only missing recovery metadata after current admitted-input and checkpoint binding checks. - Document the evidence and execution-authority boundary. ## Verification - Positive regression: red before the owner change and green afterward. - `pnpm exec vitest run server/src/services/native-runtime/native-controller-migration.integration.test.ts server/src/services/native-runtime/native-controller-recovery.integration.test.ts server/src/services/native-runtime/native-harness-backup-storage.integration.test.ts`: 81 checks passed across migration, recovery and durable storage. - The checks cover storage failure, repeated migration, exact file restoration after old-home deletion, unchanged history and ownership, and refusal to treat the archive as executable recovery authority. - A second regression reproduced the archive fence refusal for an observed failed source with a valid stopped-remote receipt. Negative controls cover missing, foreign, old and revoked receipts, live leases, pending results, retry deadlines and task ownership. Revoking the receipt during upload prevents publication. - Two additional regressions cover a follow-up queued before the prior turn finished and a remote stop before a later logical failure. Both fail with the earlier timestamp comparisons and pass with execution-start ordering. Genuine execution overlap and old receipts stay blocked. - Two historical source snapshots and controller-file hashes matched in a read-only live-versus-copy check. The first candidate reached the archive fence and exposed the observed-coordinator case. That failure is retained. A fresh candidate replay is required for the expanded proof. It does not qualify the full packaged migration or hosted Railway handoff. - Existing typed-authority checks cover stale and missing supervisors, repeat migration, byte-identical checkpoint rows, unchanged snapshots/owners/history, old-home deletion and ordinary queued-turn recovery. Foreign Runner, session and snapshot controls remain blocked. - Handoff checks cover an exact destroyed successor, a stopped successor, a broken proof and a competing tip. - Isolated replay of the two retained typed-checkpoint sources imported only derived recovery metadata. Checks confirmed unchanged checkpoint rows (12 and 9 parts), history, snapshots, finalization owners and task-session pointers. The replay used three source-module overlays with a blocked provider network; it does not qualify the released bundle. - Final-head CI attempt 1 passed typecheck, build and the migration suites but failed one unrelated budget-concurrency assertion in `decision-models.test.ts` (one provider call observed instead of two). The original log is retained. The isolated local comparison cannot classify the timing failure because this checkout has stale linked budget schema exports. The one retry of that failed CI job passed, and final-head CI attempt 2 is green. The original failure remains retained; its cause is not established. - Full local typecheck, repository tests and build are not claimed. The server typecheck reports pre-existing linked declaration mismatches outside the changed files; the full clean CI typecheck has passed. This checkout uses linked generated declarations; clean CI is required before merge. Prior head |
||
|
DottaandPaperclip
|
4740d251b7 |
fix(onboarding): retain plan approval across follow-up decisions (#19043)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Onboarding asks a person to approve a saved plan before execution tasks can start. > - A person can then approve a separate proposal for the first execution step. > - The creation gate checks only that latest card for the plan target. It loses the earlier approval of the unchanged plan. > - Decomposition and native context also omit checkbox plan confirmations. > - This pull request validates the latest execution decision and the latest plan decision together. > - The benefit is that approved work can start while human, revision, and proposal gates remain effective. ## Linked Issues or Issue Description Refs #17914, #18160, and #18283. The related-PR search found no duplicate repair. **What happened?** After a person accepted a checkbox plan card and a later execution confirmation, native task creation and REST child creation returned 403. Accepted-plan decomposition returned 422 because its lookup recognized only standard confirmations. Native task context returned no accepted plan for the checkbox decision. **Expected behavior** The latest execution proposal can use an independently approved, unchanged plan. A pending or rejected latest proposal must still stop execution. The current plan must retain valid human approval of its exact revision. **Steps to reproduce** 1. Save a plan on a server-created onboarding task. 2. Create a revision-bound checkbox confirmation and accept it as a human. 3. Create and accept a separate, untargeted confirmation for the first execution task. 4. Create the child through the native tool or REST API. 5. Separately, try accepted-plan decomposition or read native task context after step 2. **Paperclip version or commit** The workspace's registered hosted release is Core `09c05c55050fede448196637cbaa543a154da76e`. The three regressions also failed against local source `3cdb367369f29692b81cf19b514a2abdba90734f`. **Deployment mode** Hosted native Runner with ACPX Claude. Local regression proof uses actual PostgreSQL, authenticated HTTP routes, native tool authority, and synthetic task data on macOS arm64. No customer data or credentials are in the fixtures. ## What Changed - Validate the newest ordinary proposal and the newest plan proposal separately. Keep the existing human-card and conversational provenance checks for each. - Require the accepted plan target to match the current document and revision. A newer pending, rejected, invalid, or foreign plan decision cannot fall back to an older approval. - Share persisted confirmation parsing across admission, decomposition, and native context. Validate checkbox selections against the saved options and limits. - Return `acceptedPlan.approvedOptions` with the user's saved selected IDs and option text on later wakes, including revision-pinned context. Keep unchecked defaults out of scope. - Add behavioral and safety regressions. Document the approval rule. ## Verification - Three behavioral regressions failed before the repair: child creation returned 403, decomposition returned 422, and native context returned `acceptedPlan: null`. All three now pass. - Two scope-retention regressions failed before the review fix: native context omitted the selected option after a later approval, with and without a pinned plan revision. Both now pass. Unchecked defaults stay excluded. - Current admission, native-tool, and accepted-confirmation replay suites: 131 tests passed. Existing first-task completion safety checks: 17 passed, including the atomic decision recheck. - Safety cases cover missing and rejected plan approval, agent-only decisions, foreign source runs, changed revisions, invalid checkbox selections, newer pending or rejected proposals, and task-creation replay. - Full repository `pnpm -r typecheck` and `pnpm build` passed locally on final head `7c0e8ab784`. - [Full final-head CI](https://github.com/paperclipai/paperclip/actions/runs/41911580531) passed: all Vitest shards, Runner checks, eight browser E2E shards, build, release registry, and canary dry run. The required `verify` and `e2e` aggregates passed. Storybook checks were skipped for this backend change; the file-selection job was skipped because full CI ran. - Earlier local typecheck attempts exposed Copilot verification during build preparation, stale declarations across the rebase, and two test-fixture typing errors. The assets were verified, declarations rebuilt, and fixtures corrected before the complete final-head pass. - Full local Vitest attempts were interrupted for the rebase and review repair. The duplicate final local run was stopped after full Linux CI passed. No complete macOS Vitest-suite pass is claimed. - A fresh Claude-provider acceptance run remains unverified: the local Claude account is signed out and no Anthropic API credential is available. The actual native tool and HTTP paths are verified with real PostgreSQL persistence. This PR does not deploy or recover the reported production task. - Greptile: 5/5 on `7c0e8ab784`, with the review thread resolved. ## Risks A later acceptance must not revive rejected or changed scope. Admission validates both current decisions and retains source, company, human provenance, and document locks. The extra plan-proposal lookup is bounded and limited to accepted, untargeted first-task proposals. Ordinary delegation adds no new proof lookup. The native context adds an optional `approvedOptions` projection. Revision-pinned context now performs the existing scoped approval read to recover that selection. No database schema, request, provider, or background worker changes are included. ## Model Used OpenAI Codex, exact model `gpt-6.1-sol`, max reasoning effort, with repository, shell, browser, and GitHub tools. The current session metadata confirms the model and effort. The context window size is not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Dotta
|
8002dfb06e |
fix(evals): repair bounded TestOps protocol admission (#19011)
## Thinking Path > - Paperclip lets people manage AI agent work through a governed control plane. > - Runner protocol qualification uses real providers and records selected cases in TestOps. > - A manual registration run starts an Allure watch session before it validates the case selection. > - Watch creates the job run, but it does not fill the empty job-run input in the child environment. > - The selection helper therefore stops before it can validate the new job run. > - A configured AWS concurrency ceiling can also reject a valid smaller hosted-runner request. > - Hosted tool-cache permissions can also violate Runner qualified-executable admission before a provider starts. > - This pull request captures official session identities, keeps both concurrency ceilings, and stages the same Node bytes in an owned directory. ## Linked Issues or Issue Description Refs #18985. **What happened?** The [first registration attempt](https://github.com/paperclipai/paperclip/actions/runs/41877603952) created an Allure job run, then stopped with `runner_testops_jobrun_missing`. The child inherited an empty `ALLURE_JOB_RUN_ID`. The helper made no job-run API read and executed no provider case. The current configured concurrency ceiling is 100. The controlled hosted cap is 57. The original catalog guard rejects 100 before it applies a valid requested value of 2. The [first service-selected attempt](https://github.com/paperclipai/paperclip/actions/runs/41884365232) reproduced this second pre-provider failure. Its selected case was correctly reported as broken. No provider cell ran. A [second service-selected attempt](https://github.com/paperclipai/paperclip/actions/runs/41887671585) passed selection, lock and cost preflight. Runner rejected a qualified ACPX launch artifact as group- or world-writable. It recorded no model request. The verified sidecar is mode 0755. Hosted Node is the only other launch artifact, which points to Node as the cause. The original run did not retain its exact mode. The TestOps reporter retained the infrastructure failure. **Expected behavior** The manual path must capture the job-run and launch IDs from the active official watch session. It must validate those IDs against the same GitHub run, job, launch and standing plan before provider work. A requested concurrency value must fit the configured ceiling and the selected runner cap. The default must use the lower ceiling. Before credentials, the worker must stage the same Node bytes in an owned directory and preserve Runner permission and digest admission. **Steps to reproduce** 1. Dispatch the trusted Runner protocol workflow with reporting enabled and an empty job-run service input. 2. Let `allurectl watch` create the registration session. 3. Observe that the original helper rejects the empty child input before selection validation. ## What Changed - Call the supported `allurectl job-run env` command for manual registration. Use a fixed argument list without a shell and bound its time and output. - Parse only numeric job-run and launch IDs. Reject missing, duplicate, malformed or conflicting values. Discard all other command output. - Bind the captured launch ID to the authoritative job-run response. Keep the service-supplied ID path and its existing checks. - Apply requested parallelism within both configured and physical ceilings. Use their minimum as the default. Keep the selected-case capacity and provider budget checks. - Snapshot the same current Linux Node bytes into a fresh owner-only directory before credentials. Verify the copy, execute it to bind its path and version, and put only that directory first on the job-local path. Record original and copied Node modes and hashes plus the actual sidecar mode and hash. Keep shared cache permissions and Runner security checks intact. - Add focused context, identity, concurrency, permission and secret-exclusion controls. Supply the production physical cap in the existing release-workflow fixture. - Fix the existing Kimi cancellation fixture to await the complete readiness sentinel within the same 100 by 10 ms bound. Keep its cancellation and cleanup assertions. ## Verification - The original empty-ID failure reproduces locally on the actual helper with all 41 standing cases and zero job-run API reads. - All 63 Node publication, projection, metrics, permission and workflow-security tests pass. The repaired manual context flows through the actual selection helper to case 194. Wrong launch, duplicate IDs, malformed values and command failures still stop selection. - The original catalog prefix rejects configured 100 / hosted 57 / requested 2. The repaired prefix admits 2, clamps a missing request to 57, keeps legacy 100, and rejects requests above either ceiling. - The full release-policy CI command passes locally: 72 release-workflow tests and 107 companion tests. The first sandbox run could not bind its loopback fixtures; the same tests passed with local-network access. The prior-head CI fixture omission is retained and fixed by supplying the same physical cap as production. - Both full static Node file groups pass locally: 418 passed and 11 existing opt-in skips, at concurrency 2. Initial runs used stale dependency links and failed. Correcting only isolated ignored links to declared cached dependencies and pnpm 9.15.4 removed those setup failures. Local Node 26.4 differs from CI Node 24; the exact-head CI below provides canonical verification. - `actionlint .github/workflows/runner-protocol-live-evals.yml` and `git diff --check` pass. - The new staging helper passes all four tests as a nonroot Linux user with networking disabled. An actual retained Runner comparison reproduces the qualified-artifact rejection with Node mode 0777. The same Node bytes at 0500 in a 0700 directory pass that launch boundary and reach a closed provider session. The local turn still fails without credentials or network; this is permission proof, not a protocol case pass. Local Linux Node 24.21 differs from the hosted 24.22.1 executable. - Exact final head `8e06ee5611` [repository CI](https://github.com/paperclipai/paperclip/actions/runs/41892896405) completed successfully. Typecheck, build and all four Runner lanes passed. Runner package tests passed 4950 tests in 297 files, with 10 existing skips. Fresh exact-head Greptile review is 5/5 with no actionable findings or open threads. Prior `59378f` CI Kimi empty-sentinel race and canceled chat worker remain retained. - The patched manual workflow has not run against live TestOps. A separate service-triggered probe supplies its own ID. The service probe used the original concurrency guard with a temporary configured ceiling of 57. After the probe, the original repository ceiling 100 was restored with no active campaigns. That probe does not verify this patched concurrency code. The next launch waits for the protected merge and source/lock verification. ## Risks An incompatible official CLI output fails closed. The helper never evaluates the output or records its unknown variables. The existing job, workflow URL, source, plan and provider checks remain required. An omitted concurrency request now uses the lower of the configured and physical ceilings. Explicit requests above either ceiling still fail. The worker now executes a private copy of its existing Node binary. A copy, mode, hash, path, version or sidecar mismatch stops before credential access. The copy does not make dynamically linked Homebrew Node portable; this workflow path is Linux only. This change does not alter the provider roster, policy, limits, retries or result grading. ## Model Used OpenAI Codex, GPT-6. The exact hosted model ID and context window are not exposed in this session. Used reasoning, code editing, repository inspection and tool execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge |
||
|
Nicky LeachandClaude Opus 5.5
|
be20d4ad97 |
fix(projects): keep projects on the storage chosen at creation (#19036)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A project stores its work in Core/local storage or in a storage plugin > - Today, Core finds the installed default each time it reads a project > - So installing a default storage plugin moves existing projects, without a saved change > - Agents that cannot reach the new storage then lose access to new workspaces > - This pull request makes Core apply the default once, when it creates a project > - The benefit is that a project stays on its storage until a person moves it ## Linked Issues or Issue Description No issue exists. This describes the problem. I searched open pull requests for related work and found no duplicate. **What existing behavior does this improve?** How Core chooses the storage for a project when a plugin claims `defaultForProjects`. **Current behavior** Installing such a plugin changes the storage of every existing project with no explicit choice. Disabling the plugin changes it back. Nothing is saved, so the project has no fixed owner. Execution environments and storage are separate contracts, so an agent can lose access to new workspaces on a project that it used before. **Proposed behavior** Core saves the storage choice on the project when it creates the project. A project with no saved choice uses Core/local storage. A plugin-backed project stays attached to its plugin. A Core/local project can move to a plugin. **Reason and benefit** A project stays on its storage until a person moves it, so installing or disabling a plugin does not change where work goes. **Alternatives considered** - Keep the default at read time and add a process that saves a choice on existing projects when a plugin arrives. This adds a background step and a race with new projects. - Add a new database column. The existing policy field already holds the choice, so a migration is not necessary. ## What Changed - `resolveProjectWorkspaceProvider` reads only the saved choice. It no longer applies an installed default. - `withDefaultProjectStorage` applies the default when Core creates a project, including a personal private project. With more than one default, a project with no explicit choice is refused. This is the existing conflict rule, now applied at creation. - `projectService.update` refuses a change of storage on a plugin-backed project. An update that omits the field keeps the attachment. - Plugin lifecycle delivery sends project events to a storage provider only for projects attached to it, and for deleted projects. Core acknowledges the other events for that provider. Other plugins continue to receive all project events. - The project settings control has no "Inherit" option. It is locked on a plugin-backed project. - `ProjectWorkspaceProviderConfiguration.source` no longer has the `"default"` value. - A project with no repository or folder ignores its saved storage choice when a task runs. A storage plugin cannot manage this type of project, so its tasks use a plain task directory. Without this change, the second run of a task reset the agent session, and remote agents could not use an empty task directory. - The plugin SDK README and the plugin authoring guide describe the new rules. ## Verification - `npx vitest run server/src/__tests__/project-workspace-provider.integration.test.ts server/src/__tests__/plugin-lifecycle-delivery.test.ts server/src/__tests__/plugin-workspaces.integration.test.ts packages/plugins/sdk/tests/workspace-provider.test.ts`: 93 tests pass. - `npx vitest run src/components/ProjectProperties.concurrency.test.tsx` in `ui/`: 9 tests pass. - `pnpm --filter @paperclipai/server typecheck` and `pnpm --filter @paperclipai/ui typecheck` pass. - `npx vitest run` on `heartbeat-session-preparation`, `workspace-preparation`, `remote-workspace-source`, `project-workspace-provider.integration`, and `plugin-workspaces.integration` in `server/`: 231 tests pass. - I did not run the full test suite locally. CI runs it. - I did not test this change on a running instance. ## Risks - **Behavior change at upgrade:** an existing project that uses a plugin only through the installed default, with no saved choice, moves to Core/local storage after upgrade. Its plugin files stay in place, but new work does not use them until an operator sets `storageProviderPluginId` on the project. This pull request has no automatic backfill. - **Missed lifecycle events:** a provider does not receive events for a Core/local project. If the project moves to that provider later, the update event starts its reconcile. - **One-way move:** a person cannot move a plugin-backed project back through the API or the UI. This is intended. If a plugin is disabled, the project fails closed, as it does today for an explicit choice. - This pull request does not stop a person from assigning an agent to a project with storage that the agent cannot reach. That is follow-up work. - This changes core plugin behavior, and I did not discuss it in `#dev` first. Maintainers can tell me if they want a discussion before review. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), through Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
8a0e7ab511 |
fix(runner): preserve ordinary append for unscoped governed terminals (#19044)
## Thinking Path > - Paperclip manages work done by AI agents. > - A native turn can park on a saved human question or approval. > - Its terminal event can return a committed receipt before observers finish. > - That option requires an explicit turn ID at the event port. > - The protocol also permits events without a turn ID. > - The runner requested the option for those events, which the port rejected. > - This change keeps those events on the ordinary append path. ## Linked Issues or Issue Description Refs: #19025. **What happened?** The native governed-wait observation can accept a schema-valid `item.completed` without a turn ID. If the later terminal also has no turn ID, the runner compares two absent IDs and requests `deferPostCommitObservers`. The event port rejects that option. The run then fails with `governed_wait_provider_terminal_unproven`. **Expected behavior** Only a terminal with an explicit turn ID can request the early receipt. An unscoped terminal must keep the ordinary append behavior and await its observers. The event port must retain its turn identity check. **Steps to reproduce** 1. Create a pending human-only interaction from the run. 2. Pass a schema-valid completed tool event without a turn ID through the real governed-wait observation. 3. Supply an interrupted terminal without a turn ID during the cancellation drain. 4. Run the actual native runtime with the PostgreSQL event port. The parent code rejects the deferral request and reports a failed terminal. **Paperclip version or commit** Reproduced on `4ea53f9eb5a24de10db3dfda18456132987c4af8`. The follow-up was identified by a review published after #19025 merged. No observed provider emission or original-task recovery is claimed. I checked current master and related PRs before this correction. **Deployment mode** Built from source. The regression uses real PostgreSQL and a deterministic provider boundary; it makes no model call. ## What Changed - Require a truthy terminal turn ID before the runner requests deferred observers. - Keep ordinary append behavior, the port identity check and all timeouts unchanged. - Exercise the production governed-wait observation with a persisted pending interaction and the real event port. - Retain exact-turn observer and checkpoint controls. Add a direct rejection control for unscoped deferral. - Document that unscoped terminals await their observers. ## Verification - First meaningful red: the actual observation, native runtime and PostgreSQL port fail on the unmodified parent runtime. The new case reports `NativeProviderTerminalFailure` in 143ms; this is not a setup or deadline failure. - Focused regression and authority controls: 12 passed, 69 filtered. Full runtime suite: 203 passed. Full PostgreSQL port and timing suites: 84 passed. - The tests leave the human-only interaction pending with continuation policy `none`, accept one result, and start only one turn. Existing scope, invalid evidence, checkpoint, abort, duplicate and restart controls remain. - Exact candidate: `8840ea2c9c308e5a60423e67fb3612d24029ba84`. Repository-wide `pnpm -r typecheck`, `pnpm build` and `pnpm check:module-boundaries` passed on this exact head. `pnpm test:run` is still in progress and is not reported as passing. Current-head CI passed: 55 success, 3 skipped and 1 neutral, including required `ci / verify` and `ci / e2e`. Greptile and Ponytail both reviewed this exact head at 5/5 with no blocking findings. The native review was source-only. No unresolved review threads remain. - No UI, database schema, agent permission or mapped user entry point changes. No deployment, live retry or provider replay was performed. ## Risks Unscoped terminals retain the existing observer wait and can still fail if it exceeds the existing drain deadline. They do not gain an early receipt. The change does not supply a missing turn ID or weaken source, company, run, session, cancellation or transaction checks. A PRP terminal is not physical-stop or accounting-completion proof. ## Model Used OpenAI `gpt-6.1-sol`, with code and shell tools. The host does not expose the reasoning mode or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
c99bcd89fe |
fix: preserve Product TestOps evidence and CI readiness (#19000)
## Thinking Path > - Paperclip manages agent work through governed product workflows. > - Product E2E tests check those workflows through the app and Runner. > - A trusted reporter sends retained results to TestOps. > - Free-form metadata and a short upload observer can lose safe, valid evidence. > - UUID navigation can also discard task metadata before the canonical query returns. > - Hermetic CI files start child processes that need CPU capacity within their readiness limits. > - This change closes the reporting contract, retains authenticated core aliases, and reserves CI CPU capacity. > - Original case identities, conditions, assertions, verdicts and deadlines remain intact. ## Linked Issues or Issue Description Refs #18983. This addresses its late [metadata disclosure finding](https://github.com/paperclipai/paperclip/pull/18983#discussion_r4470789114) and [upload readiness finding](https://github.com/paperclipai/paperclip/pull/18983#discussion_r4470790366). **What happened?** The reporter accepted model IDs and matcher names through syntax filters. A secret-shaped string could pass those filters. First-task results also use their top-level model field for an observed alias. Treating it as configured metadata can reject valid retained evidence. The upload observer stopped after 22 seconds of waits. [Required CI on the earlier head](https://github.com/paperclipai/paperclip/actions/runs/41886885193) also found two real browser title failures and five native package readiness failures. A deterministic control reproduced metadata loss during UUID-to-identifier navigation. A separate fixed-resource control reproduced startup deadline failures with four concurrent hermetic fixtures, while one and two fixtures passed. **Expected behavior** Reject unknown transmitted metadata before creating upload files. Preserve every original matcher verdict and attempt receipt. Observe server processing without rerunning the product. Keep authenticated task metadata visible through canonical navigation. Reserve capacity for hermetic child processes without changing the original readiness deadlines. **Steps to reproduce** Use a secret-shaped model ID or cleanup state in retained evidence, return an observed first-task model alias, or let upload processing settle after 40 seconds of virtual time. Hold the canonical core query pending after an authenticated UUID read. For the CPU control, run the same providerless startup/recovery fixture alone, four times concurrently, and twice concurrently under a fixed 4-vCPU quota and its original 8-second readiness limit. ## What Changed - Bind configured model receipts to each exact declared case model. Four first-task cases use captured agent configuration; their observed aliases and fallback text remain digest-only. A missing capture stays null / Not recorded. The other 16 cells use their configured profile model. Reject wrong or unknown configuration before any upload file exists. - Keep the existing model distribution: 18 Codex cells on `gpt-5.6-sol`, configured-repository case 9 on its existing `gpt-6.1-sol` override, and task-workspaces case 54 on `openrouter/deepseek/deepseek-v4-flash-0731`. The repository override exists at the [original canary source](https://github.com/paperclipai/paperclip/blob/86814c6550f52e99ede150a883bdc773cbb540e3/tests/runner-e2e/catalog.ts#L1148). No model configuration changes. - Allow only existing Matcher kinds, closed failure classes, and cleanup states `passed`, `failed`, or `not_started`. Publish each matcher's kind, ordinal, original boolean verdict and SHA256 path receipt. Full conditions remain in the linked original workflow artifact; raw dynamic path bytes do not enter TestOps. - Observe the same upload for at most 55 seconds of poll waits. Product scenarios, provider retries and scenario deadlines do not change. - Retain task core data under only the authenticated response's own UUID and identifier core keys. Preserve existing canonical data. Keep arbitrary references and full-detail keys empty until their own reads. - Retain a first-failure trace for the existing Cloud recovery test. Keep its assertions, timeout, retries, mock data and artifact policy unchanged. This adds diagnosis evidence; it does not claim to fix that separate Cloud title timeout. - Reserve half the available CPUs for Runner package CI workers, with the existing upper cap of four. A 4-vCPU host uses two workers; an 8-vCPU host still uses four. Local defaults and test selection do not change. ## Verification - Reporting controls first reproduced the two original gaps. All 14 Node contract tests now pass. Unknown configuration creates neither a result directory nor an export summary. A model valid for a different selected case is also rejected. Original functional failures and earlier accounting receipts survive. - The catalog compatibility test passes for all 20 selected cells, exact model tuples, real pure grader outputs and actual `projectFirstTaskResult` shapes. It covers observed aliases, fallback text, absent evidence and wrong or secret-shaped configured metadata. These are contract controls, not live journey passes. - Virtual-clock controls accept a correct result that settles at 40 seconds and reject a persistently wrong verdict after 55 seconds. They make no product or provider calls. - Two new core-cache controls failed before repair. The query/cache suites now pass all 8 tests. They hold canonical refresh pending and verify title/company retention, newer canonical data, full-detail enrichment and unrelated keys. UI typecheck and token gates pass. - The unchanged exact UUID cross-company link/upload browser test passed once after repair: 5.8-second scenario, 32.0-second complete command. The unchanged exact Cloud task-entry diagnostic passed once: 5.4-second scenario, 23.3-second complete command. Its original canonical-only timeout remains unexplained. All five Cloud tests still collect after the trace setting. - The providerless Linux control used the retained Runner and compiled controller from source `3a2249dd`, with a fixed 4-vCPU quota and original 8-second readiness limit. Solo passed 1/1; four concurrent fixtures failed 4/4; two concurrent fixtures passed 2/2. At each failure, Rust had accepted command 2 but startup was still pending; checkpoint and published status agreed, one connection was authenticated, no replay backlog existed, and the sidecar had not launched. Passing controls preserved the expected startup failure through controller replacement. No network or credentials were available. - That control uses Linux amd64 emulation and Node 24.21; hosted CI used Node 24.22.1. It supports the worker-cap change, but does not prove that the original `m5n.xlarge` host caused every CI failure. The original red attempts remain recorded. - Actual config-load controls verify 1 CPU → 1 worker, 2 → 1, 4 → 2, 8 → 4 and 16 → 4, with no local, timeout or retry override. The isolated policy change preserves the same 297 test-file inventory; integration of current master `89703c3ff7` retains all 298 current native package files. The original PR package/native command plans remain unchanged. - Final head `877e28da32f2d2f0d12906c89aff085e2c6ea943` passes both required checks, `ci / verify` and `ci / e2e`, in [CI run 41897677652, attempt 2](https://github.com/paperclipai/paperclip/actions/runs/41897677652/attempts/2). The native package passes 4,950 tests with 10 existing skips. All eight browser shards pass; the Cloud shard passes all 41 tests, including all four unchanged recovery cases. - Attempt 1 preserved 46 successful jobs but its one-file heartbeat recovery shard hit the original 20-minute job limit without a named assertion or completion verdict. Its cause remains unidentified. One explicitly bounded debug replay retained those 46 successes and ran only that leaf and its dependent rollup. The unchanged file passed all 412 tests (346.91-second file, 371.34-second Vitest command, 7m11s job). Source, cases and deadlines did not change. This establishes intermittency, not a causal repair. The original timeout and debug pass both remain recorded. - Fresh [Greptile 5/5](https://github.com/paperclipai/paperclip/pull/19000#issuecomment-6637604918) and [Ponytail 5/5](https://github.com/paperclipai/paperclip/pull/19000#issuecomment-6637489454) reviews name exact head `877e28`. The single review thread is resolved. Live TestOps bootstrap and complete 20-case acceptance remain held until reviewed delivery. CI and focused controls do not count as product plan passes. ## Risks - Future model or Matcher kind changes require a reviewed reporting-contract update. - TestOps shows a closed Matcher kind and ordinal. The path digest binds the original condition in retained evidence. - Core alias data retains the existing authenticated query-client and server company boundaries. It never becomes full-detail evidence. - Fewer CI workers can increase one job's duration. The full suite must verify the final policy. The paid evaluation matrix and its parallelism do not change. - The original Cloud title timeout and the first-attempt heartbeat CI hang do not have proven causes. Current unchanged assertions pass, but this does not establish repeatability. The Cloud test now retains its first-failure trace. A successful upload or focused control alone cannot establish a complete product pass. ## Model Used OpenAI Codex with reasoning, tool use and code execution. The host does not expose the exact model ID or context window to this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
513c03cb3e |
fix(e2e): configure supported rolling agents with warm policy (#18994)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Rolling tests must exercise the lifecycle they claim to cover.
> - The warm lane selected a warm environment.
> - Its profiles also set an explicit per-turn agent policy, which took
precedence.
> - The test then waited for a live workspace that per-turn cleanup had
correctly retired.
> - This pull request sets warm agent policy for providers admitted by
the production warm capability.
> - The matrix now checks effective lifecycle and preserves per-turn
coverage for every qualified native profile.
## Linked Issues or Issue Description
Builds on merged #18982 for the source controller dependency link. This
PR now targets master.
**What happened?**
The warm rolling fixture built a Codex agent with `lifecycleMode:
per_turn` and an environment with `runnerLifecycleMode: warm`. The
explicit agent policy won. The initial task completed, its disposable
workspace retired, and the fixture waited for a live file before it
reached deployment.
**Expected behavior**
Warm rolling, legacy migration, remote-file and controller-outage cells
must create actual warm agents. Only providers supported by the
production managed warm-session capability belong in that lane. All
qualified native profiles retain per-turn coverage.
**Steps to reproduce**
Build the old warm Codex cell's agent and environment and compare their
lifecycle policies. The new effective-policy regression fails on the old
catalog and passes on this branch. No duplicate exact-scope warm-fixture
PR was found.
## What Changed
- Share the production managed warm capability filter and explicit warm
agent policy across rolling, legacy migration, remote-file and
controller-outage suites.
- Check effective agent and environment policies and update matrix
coverage/counts.
- Read completed warm turns from the exact saved snapshot while
independently requiring the retained, bound workspace location.
- Keep both rolling lanes separate from the outage simulation and
exercise their assembled supervisor requests.
- Version the corrected definitions and keep every qualified native
profile in the per-turn lanes.
## Verification
- Baseline regression: failed against the unchanged catalog; retained
before the fix.
- Fixed catalog/controller selection: 39 checks passed.
- The first real warm-policy attempt completed and saved its workspace
but timed out because the file oracle demanded live content; retained
trace responses prove the API returned the correct saved snapshot with a
retained lease.
- A new regression fails on the previous reader and passes after
separating saved file source from retained versus released location.
Focused catalog, reader and rolling checks: 81 passed.
- The next warm attempt reached replacement and exposed a second fixture
defect: warm also selected a SIGKILL outage, which the supervisor
rejected alongside rolling mode. The failure remains retained; its four
owned sandboxes, temporary key and local bucket were deleted.
- An outage-selection regression failed before the fix; 57
outage/supervisor checks pass afterward, including assembled requests
for both rolling lanes. The fresh warm Codex run passed all 56 checks in
8.5 minutes: original run/session/runner/lease retained, authenticated
takeover, both predecessor homes removed, later native continuation and
saved file readback. Two replacement receipts recorded 742 ordinary
reads and 75 writes, all 75 read back, with zero HTTP probe failures. A
transient `native_workspace_sync_out_failed` phase recovered before
successful finalization and remains retained. This source-lane result
does not claim released-image or hosted Railway qualification.
- The invalid earlier attempt was interrupted before deployment. Its
four archived disposable sandboxes were deleted and verified absent; its
temporary key and local bucket were removed.
- Broad local typecheck reached stale shared database declarations in
this checkout's linked build dependencies. The fixture's own type error
was corrected; clean-head CI remains required. No full build is claimed
locally.
- An incidental report-catalog test selected by a substring filter
failed to generate its normalized output. That failure is retained
separately; the focused catalog/controller checks pass.
- Review follow-up: fixed the same policy conflict in the base
file-storage, legacy migration and controller-outage suites. The roster
now has 822 valid cells; unsupported warm combinations remain covered
per-turn. Updated the explicit supervisor-request control to use a
supported warm provider. Focused catalog, file-reader, migration,
rolling and supervisor checks pass (133 checks).
- Earlier full CI passed at
|
||
|
|
4ea53f9eb5 |
fix(server): resolve issue identifiers in file-resources routes (#16352)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents and the board read task workspace files through the
`/api/issues/:issueId/file-resources/*` routes
> - Clients can identify a task by its UUID or by its human identifier,
for example `ABC-230`
> - The issues router changes identifiers to UUIDs, but the
file-resources router does not
> - Thus an identifier goes into a `uuid` comparison, Postgres rejects
the query, and the route returns 500
> - This pull request resolves identifiers in the file-resources router
and returns 404 for malformed ids
> - The benefit is that identifier links work, and bad ids give a
correct 404 and no server error
## Linked Issues or Issue Description
**What happened?**
A request to `GET /api/issues/ABC-230/file-resources/list` returned 500.
The server log showed `invalid input syntax for type uuid` from
`workspaceFileResourceService.getIssue`. Sentry recorded this error on
hosted staging.
**Steps to reproduce**
1. Create a task that has an identifier, for example `ABC-1`.
2. Send `GET /api/issues/ABC-1/file-resources/list?workspace=project`.
3. The server returns 500. The same request with the task UUID returns
200.
**Expected behavior**
The route accepts the identifier, as the other `/api/issues/:id` routes
do. An unknown or malformed id returns 404.
## What Changed
- Added `resolveIssueId` to `WorkspaceFileResourceService`. It changes a
task identifier to the task UUID. It does not check visibility.
`getIssue` still does the privacy-scoped read.
- Added a `router.param("issueId")` handler to the file-resources
router. It calls `resolveIssueId`, the same as the issues router does
for its routes.
- `getIssue` now returns 404 before the query when the id is not a UUID.
This stops the Postgres cast error for all callers.
- Added a regression test for identifier, lowercase identifier, private
task, unknown identifier, and malformed id on the `content`, `list`, and
`availability` routes.
- Updated the test service fakes with the new method.
## Verification
- `pnpm vitest run src/__tests__/file-resources.test.ts
src/__tests__/file-resources-git-scan-load.test.ts` in `server/`: 55
passed.
- Without the source change, the new test fails with `expected 200 "OK",
got 500 "Internal Server Error"`. This reproduces the production error.
- I did not run a full local `tsc` because the sandbox did not have
sufficient memory. CI runs the typecheck.
## Risks
- Low risk. The change adds one indexed lookup on `issues.identifier`
(unique index) only when the path value has the identifier format.
- The identifier lookup does not check visibility. `getIssue` then
applies the privacy-scoped read, so a private or unknown task gives the
same 404 as before. The route does not show if a task exists.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edit, test runs).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Paperclip <noreply@paperclip.ing>
canary/v2026.1104.0-canary.13
|
||
|
|
3cdb367369 |
fix(acpx): report how the sandbox agent process session ended (#16366)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Hosted `claude_local` runs use the ACPX engine. The agent process
runs in a sandbox, and a host-side process session bridge carries the
ACP stream.
> - Some hosted turns fail with only "The connection to Claude was
lost." (`acpx_turn_failed`, terminal session category `connection`). The
failure report shows one pending `execute` tool. It does not show why
the agent process went away.
> - The bridge already records how the session ended: the remote exit
code and signal, or a host proxy close. But it writes that evidence only
to the run log. The run failure report never receives it.
> - Without it, we cannot tell an OOM kill (`SIGKILL`) or a provider
process exit from a bridge or proxy close. Each cause needs a different
fix, so we must not guess.
> - This pull request keeps the first terminal evidence on the bridge
handle and copies it, as closed values, into the run failure report's
execution context.
> - The benefit is that the next recurrence names the cause, and the
correct fix can follow.
## Linked Issues or Issue Description
**What happened?**
Hosted `claude_local` runs on staging and production failed with "The
connection to Claude was lost." One run failed about 4 minutes in, 7–10
seconds after its last ACP event. Another failed after about 3.5 hours
with no ACP events for most of that time. Both had one pending `execute`
tool. The report had no information about how the sandbox agent process
ended.
**Expected behavior**
The run failure report says how the sandbox agent process session ended.
Examples: the remote process exited with a code, the remote process was
killed by a signal, or the host proxy closed.
**Steps to reproduce**
1. Start a hosted ACP `claude_local` run on a sandbox target.
2. Kill the agent process in the sandbox while a tool runs, for example
with `kill -9`.
3. See that the run log has an `ACP process session terminal:` line, but
the run failure report has no matching field.
**Paperclip version or commit**
`master` at
|
||
|
|
53735db7de |
fix(retry): resume verified preparation failures (#19004)
## Thinking Path > - Paperclip must preserve saved agent work while proving earlier execution stopped. > - A deliberate task Retry can observe a newer failure while an older preparation still holds execution. > - That older preparation can also have a finalized empty object log despite never reaching the provider. > - Binding only the latest run and requiring every log field to be absent keeps valid recovery deferred. > - This change follows the exact persisted predecessor chain and binds current Retry authority to eligible holds. > - It recognizes only an exact finalized empty-log receipt within the existing strict preparation proof. > - One fresh user turn can then start with saved work and the existing ownership, stop and accounting checks. ## Linked Issues or Issue Description Refs #18989, #18860 and #18997. The first waits for predecessor cleanup during automatic dispatch. The second selects the committed native session owner. The third retains admitted Retry authority when no source recovery action exists. None passes current board Retry intent to a held failed predecessor. A search of open repairs found no duplicate. ROADMAP.md was checked; this is a recovery bug fix. **What happened?** A failed native preparation retains a legacy run row and an inherited session. Its remote workspace is later saved and verified. An automatic successor also fails during preparation. Board Retry observes that newer run, but the effective hold belongs to its predecessor. The Retry does not bind the predecessor's recovery generation. The native historical-source guard then keeps the request deferred. **Expected behavior** Carry the deliberate Retry through its verified same-company, same-task and same-agent persisted predecessor chain. Start one fresh user turn after every source passes current execution, stop and saved-work checks. Keep incomplete or unrelated proof held. Preserve prior work and recorded action outcomes. **Steps to reproduce** 1. Fail native preparation with an inherited session and a remote workspace. 2. Finish its real workspace recovery and stop receipts. 3. Save a newer failed automatic successor with `retryOfRunId` bound to that source. 4. Submit ordinary task Retry for the newer run. 5. Before the change, no successor is admitted. The request remains deferred behind the older hold. **Paperclip version or commit** The regression fails on master `039e0dad2c51c80251e1302a19c5566e039c357a`, which includes #18989. **Deployment mode** Source checkout on macOS arm64 with real PostgreSQL, HTTP task Retry and the assembled heartbeat service. The remote filesystem boundary uses temporary files. Hosted rollout and an original-task Retry are not yet verified. This PR incorporates the reviewed empty-log follow-up #19021, preserving its commits and regression controls. Both parts are required for hosted preparation recovery, so the combined revision is qualified and deployed together. ## What Changed - Accept a finalized zero-byte `s3_segments` log only when its reference binds the exact company, agent and run, its digest matches empty content, compression is false, and no output progress or nonempty excerpt exists. Add a real-recorder HTTP regression and 14 negative controls. Unknown or nonempty logs stay held. - Read a complete persisted Retry chain in one recursive query, limited to 32 runs. Reject missing, cyclic, newer, unfinished, cross-company, cross-task and cross-agent sources. Transcript hints grant no authority. - Bind eligible historical recovery records to the current Retry intent. Keep complete or running recovery budgets. Leave successful ancestors and mismatched recovery-source records unchanged. - Recheck that same chain when selecting an ancestor's fresh-user strategy. - Admit only a held ancestor through the shared native preparation proof. Keep the closed preparation failure codes, no-provider state, profile, input, result, output and accounting checks. Treat inherited session input as prior context and accept the exact bounded host start marker. Verify current ownership, saved work and stop receipts before forcing a fresh session. - Add HTTP and PostgreSQL regressions and document the rule. - Give the HTTP Retry nudge and explicit test worker the same real heartbeat facade and provider boundary. Establish the revoked catalog receipt before either can dispatch. ## Verification - Combined integration head `91f021f233151192f69dd7a035d083abcc67c47c` requires fresh complete CI and review; qualification is in progress. - Repair validation: all **six** recovered preparation provider-boundary cases pass. The catalog cases still require the exact access refusal, zero provider inputs and zero provider turns. The success case still checks preserved file bytes and one native turn. - [Previous exact-head CI](https://github.com/paperclipai/paperclip/actions/runs/41883421120/job/126785508136) failed the catalog case with `remote_workspace_transport_unsupported`. The route created a default heartbeat that could race the injected worker and allocate outside the controlled transport. The repaired test shares its real heartbeat across both dispatchers and holds the agent start lock until the revoked catalog receipt is committed. It tracks and drains both Retry nudges during teardown. It changes no production access check or expected refusal. - The first broad local reproduction hit a full local disk. It passed 201 cases and failed 86, including explicit manifest free-space errors. Generated build caches were removed from these isolated checkouts. The owning run passed 670/671 cases. Its one failure was the saved-Files loader refusing insufficient temporary disk space. After cleanup, that unchanged test passes in isolation. The final shared-facade and start-lock version passes all six dispatch cases. Both legacy continuation suites pass all 29 cases. The final fixture teardown also retires deferred input before removing recovery proof, so global queue sweeps cannot revisit earlier cases. All seven dispatch and saved-Files controls pass together after that change. No disk reserve, assertion or timeout changed. The initial local typecheck also stopped on `ENOSPC` while materializing provider assets. Server typecheck now passes after cleanup. No test timeout or reserve was lowered. - Current-head CI is complete: 52 successful checks, three expected skips, and a successful security status. Both required gates (`ci / e2e` and `ci / verify`) passed on `0adc2e38103533dd45bcc56f957ac019642c57b1`. [Full CI evidence](https://github.com/paperclipai/paperclip/actions/runs/41898305521). Greptile scored final head `0adc2e38103533dd45bcc56f957ac019642c57b1` 5/5 with no blocking findings or unresolved threads. The earlier verification below describes the previous head. - RED on unchanged master `039e0dad2c`: the certified-preparation sibling passes, while the inherited-session case admits zero successors instead of one. A second isolated RED on latest base `8f06bd4e17`, including #18997, uses the exact final three-node test fixture. The HTTP route returns 202, then the test observes zero successors instead of one. All baseline production files remain unchanged. The initial dependency/setup failures are separate from these behavioral failures. - The positive cases cover the failed-failed-successful chain. They keep the completed result unchanged, verify the original saved file and validate fresh-session continuation authority. Repeated HTTP input and concurrent resume calls keep one successor. - PostgreSQL controls accept a complete 32-run chain. A 33-run chain and a newer-created predecessor return no usable ancestry and do not bind the older recovery record. - Safety controls cover company, task and agent scope, conflicting canonical bindings, missing and cyclic edges, transcript hints, live execution and controllers, unknown stop, failed saving, unsafe writer state, Stop, actor and generation changes, independent holds and mismatched recovery records. - Ancestor controls retain provider-event, native-input, session-checkpoint, controller-recovery, result, process, runtime, accounting and unknown-result guards. Host start and warning prefixes use exact origins, sequence and payloads. A fifth extra event is rejected. - The first complete owning run found 22 introduced admission failures. The initial historical-native allowance also applied to exact-source Retry. It admitted contradictory preparation evidence. The fix now requires a canonical ancestor and the shared strict startup proof. The original direct-source assertions remain unchanged. - The next focused run passed 82 cases, including both catalogue-warning controls. One new result fixture lacked the database's required owner tuple. Two existing controls reached their original 30-second limit. The fixture now uses the real native task and completion-contract owner tuple. A controlled nine-case run passes the result case, both unchanged timed-out controls, three failure-code controls and all three positives. The earlier timeout causes remain undetermined. All failed-run logs are retained. - Complete owning suites passed on previous head `137a8eb3febf41052e112fc3c4ffe1abb92bcab7`: all 675 tests across workspace recovery, task Retry, explicit continuation and legacy continuation. The command exits successfully with normal teardown. This includes every earlier introduced admission failure and the two unchanged timeout controls. Its base includes the merged #18997 authority repair and #18982 rolling-copy packaging repair. The rebase retained both test imports at their one adjacent-import conflict. - Node 24.19.0 and pnpm 9.15.4. Module boundaries and `git diff --check` pass. The initial PostgreSQL start lacked package library links after install with scripts disabled. The documented hydration restored the isolated dependency; that setup failure was not a behavioral red. - Previous-head [full typecheck](https://github.com/paperclipai/paperclip/actions/runs/41883421120/job/126785506215) and [full build](https://github.com/paperclipai/paperclip/actions/runs/41883421120/job/126785506094) passed. Earlier attempts received synchronized runner shutdown signals and a lost-runner annotation. The shutdown initiator remains unknown. A CI owner used the documented hosted-runner failover for attempt three and restored the original routing value after capture. That attempt exposed the catalog failure above. Source workflows and check requirements remain unchanged. The current head now has green CI, Greptile 5/5, no unresolved review threads, and no merge conflicts. Hosted rollout and the original task are not yet verified. - Combined recovery proof: 24 of 24 cases passed on this exact head in 43.13 seconds, with normal teardown and unchanged assertions and limits. They cover the real recorder, all 14 empty-output rejection controls, ancestor and direct Retry, saved messages, the configured native provider, and verified stop. The targeted TypeScript graph also passed; the preceding integration step passed 95 safety/provider/message cases and all 49 task Retry cases. - The incorporated #19021 was independently red/green with the real recorder: unchanged predecessor code returned HTTP 202 but admitted zero successors. Its 18 focused cases, all 690 owning recovery cases, full typecheck/build, 1,904 local chat cases, complete CI and Greptile 5/5 passed before integration. The unchanged CLI startup lane passed one justified retry; no hosted acceptance is inferred from those results. - Additional local qualification of the unchanged workspace fixture alongside the native bootstrap fixture passed 288 of 289 cases. One manual-admission HTTP case ended with `socket hang up` after 451 ms, without an assertion or HTTP status. That unchanged case passed once in isolation at the original 40-second limit, with normal teardown, and passed in exact `0adc` CI in 17.2 seconds. The earlier disconnect cause remains unknown; no assertion or timeout changed. - An earlier CI attempt failed the newly merged native ingress fixture because its shared Node command failed strict permission qualification. The current base includes #19020's owned executable fixture repair; no production permission check was relaxed. - Integration with current master had one documentation conflict. An automatic test merge also duplicated an app declaration; that syntax failure was retained and fixed by reusing the configured real HTTP app. Both new message-state assertions remain intact. Workflows and lockfile retain master's contents. ## Risks - A damaged or overlong chain remains held. The helper does not truncate it and grant partial authority. - The new query runs only for a deliberate Retry or its ancestor strategy check. It adds no background polling, table, credential or provider policy. - A fresh intent is not stop or file-recovery proof. The existing owners retain those checks. Prior effects remain fenced, and no old result is replayed. - Local tests do not prove hosted rollout. Live acceptance must use the served fixed revision and the task's original saved work. ## Model Used OpenAI GPT-6 through Codex for the original change. OpenAI GPT-6.1 Sol (`gpt-6.1-sol`), xhigh reasoning, for the merge and CI repair. Both used repository tools and PostgreSQL verification. Context-window limits were not exposed by the harness. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Devin Foley <devin@paperclip.ing> |
||
|
DottaandPaperclip
|
22735cfcea |
fix(native): preserve governed terminal receipts during observer settlement (#19025)
## Thinking Path > - Paperclip manages work done by AI agents. > - A native turn can yield to a saved human question or approval. > - The provider must stop, and its terminal event must be saved. > - The old drain waits for both the event transaction and its later observers. > - A slow observer can make a saved terminal look unproved. > - This change separates the committed receipt from owned observer work. > - Checkpoint and result acceptance still wait for that work to settle. ## Linked Issues or Issue Description **What happened?** A native governed wait can fail with `governed_wait_provider_terminal_unproven` after its exact interrupted terminal is already in PostgreSQL. The event port waits for postcommit accounting, output and projection callbacks before it returns the receipt. Those callbacks can cross the provider drain deadline. **What did you expect to happen?** A committed terminal proves the event transaction. Its observers remain separate work. A checkpoint or result must not be accepted until those observers settle. A pending human-only question must keep its audience and continuation policy. **Steps to reproduce** Run the real PostgreSQL control-plane regression. Commit a same-turn interrupted terminal. Hold its postcommit observer beyond the existing provider drain. The parent code reports an unproved provider terminal. The fixed code retains the committed receipt and joins the observer before checkpoint or result acceptance. No overlapping repair was found in the current master or open-PR search. Related [#18977](https://github.com/paperclipai/paperclip/pull/18977) covers final conversation-response selection. It does not change this receipt boundary. The source reproduction is consistent with the observed failure. The exact hosted provider code is unavailable, so it does not prove that every observed failure used this branch. ## What Changed - Add an explicit terminal-only option to the internal event port. Ordinary appends keep their existing behavior. - Return that receipt only after the normal transaction and company, agent, run, session, source and revocation checks. - Retain the full observer promise. Join it before checkpoint and result writes. Keep its original rejection and observe it after an aborted join. - Reuse the existing provider drain and checkpoint deadlines. Preserve duplicate and restart replay without another model turn. - Document the receipt and duplicate-observer replay boundaries. Keep human-only and no-continuation cards unchanged. - Assert the actual Codex facade requests deferral only for its interrupted terminal. Isolate empty skill inventory in monitor-only process fixtures while retaining their real scheduler, lifecycle, selection snapshot and three-second deadline. ## Verification - Meaningful parent red: actual native runtime and real PostgreSQL event port. The terminal row commits, but a held observer causes `NativeProviderTerminalFailure` with `governed_wait_provider_terminal_unproven`. - Current candidate `0db24ae74ee42fdf92035d8407fc095e9f73cc6a`: full runtime suite, 203 passed. Real PostgreSQL control-plane, timing and monitor suites, 103 passed. Runner TypeScript and full layered server typechecks passed on the unchanged candidate source tree. The controls cover uncommitted and foreign evidence, explicit false and failed terminals, held and rejecting observers, abort and controller revocation, duplicate receipts, and new-port recovery. The real pending human-only card stays unanswered with continuation policy `none`. Recovery does not call `startTurn` again. - `pnpm -r typecheck`, the production server build (including the real Runner release binary), module boundaries and vendor dependency verification passed on `6d75`; the follow-up changes only tests and documentation. Diff checks pass on the current candidate. The prior `399ad3dc2b` repository-wide `test:run` was stopped as obsolete during the general-server phase after preserving its progress (exit 130, incomplete); its repository build was not reached. Its six completed chat shards passed 1,904 cases, but this is prior-head evidence and not a full-suite pass. Full current-head CI is required before merge. - CI attempt 1 on `6d75` failed one real three-second monitor-settlement assertion; the aggregate verify gate consequently failed. Other jobs passed. The actual CI merge tree was `3037bf6db162da3b0a9172046360401f8449ebc5` (`6d75` into `89703c3`). An unchanged full-file replay reproduced three timeouts. Original-deadline inspection found the legacy process still preparing; local phase timing measured cold skill inventory at 6.629 seconds. The original CI did not retain that phase detail, so its exact subcause remains unproved. The fixture correction supplies empty skill entries for these skill-free process tests and still invokes the selection-snapshot callback. It preserves all assertions and the three-second deadline, with no new drain wait. All 21 cases pass; deliberately restoring the old blocked-status rejection still fails the real blocked-check case with 409. No CI retry was issued. - The prior current-head native review was 4/5 with no production correctness finding. Its requested direct facade-options assertion and duplicate-observer wording correction are included. Current-head CI is green (55 success, 3 skipped), including required verify and E2E; current-head Greptile is 5/5 with no actionable findings. The old native advisory remains tied to `6d75`, and its requested findings are addressed. No new native assessment is claimed or required by protection. - The first draft broke three existing transaction-start timing controls. The no-debt path now keeps the original synchronous start, and all three pass. Two draft type errors were corrected. A further real-PG red found an extra default-append assertion after commit. Removing it restores the original committed receipt after a late abort; the same aborted signal still cannot write a checkpoint or result. Original setup and assertion failures remain in private verification receipts. - Normal merge of current master `3a2249dd5768313f04b2aa3860dfd363492eda16` has no conflicts. The affected production source blobs did not change in that baseline merge. - Local proof uses Node 24.22.2 on Darwin arm64, a real PostgreSQL instance and deterministic provider-boundary fixtures. It makes no model call. No deployed replay or original-task recovery is claimed. ## Risks Deferred observers must remain owned. A missed join could accept incomplete accounting or output. A rejected or stalled observer must prevent acceptance, and an abort must prevent late checkpoint/result writes. The tests exercise these boundaries. The port keeps the existing best-effort trace-publication handling. This change does not turn a PRP terminal into physical-stop or accounting-completion proof. ## Model Used OpenAI `gpt-6.1-sol`, with code and shell tools. The host does not expose the reasoning mode or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
c3715e74a6 |
perf(ci): let the public-install registry reuse warmed scoped metadata (#16838)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Every pull request runs the `ci / Build` check. This check was the slowest check in recent green runs: 432s in run 39254079691, 480s in run 39251304886 and 472s in run 39245233967. > - The slowest step in Build is "Verify built-in Grok from a clean public npm install" (162s and 207s). Its consumer `npm install` took 69s and 75s, even though a background step fills the npm cache first. > - The loopback registry in `scripts/public-install-registry.mjs` sends packages that it does not serve to npmjs with a 302 redirect. It escaped scoped names with `encodeURIComponent` (`%40scope%2Fname`). npm requests and caches the same metadata as `@scope%2fname`, and npm keys cached metadata by URL. > - Thus the consumer install could not use the scoped metadata that the warm-up had just put in the cache. It downloaded and parsed all 291 scoped packuments again (about 285 MB of full packument JSON) inside the 4-vCPU container. > - This pull request makes the redirect use the same escaping as npm. > - The benefit is a shorter Build check on every pull request. The installed bytes do not change. ## Linked Issues or Issue Description **Describe the bug** The consumer install in the public-install verifier does not use cached scoped package metadata. The warm-up step (`scripts/warm-public-install-npm-cache.mjs`) caches this metadata through direct npm requests. **Steps to reproduce** 1. Run `PAPERCLIP_PUBLIC_INSTALL_NPM_CACHE=/tmp/c node scripts/warm-public-install-npm-cache.mjs`. 2. Install the same dependency set through the loopback registry with `npm_config_cache=/tmp/c` and `--loglevel http`. 3. Count the `(cache miss)` lines. **Expected behavior** No cache misses for metadata that the warm-up already fetched. **Actual behavior** 291 cache misses. All of them are scoped packages that are requested as `https://registry.npmjs.org/%40scope%2Fname`. The cache holds them as `https://registry.npmjs.org/@scope%2fname`. Related: #16513 changes when the verifier waits for the warm-up. This pull request does not overlap with it, and the two changes add together. ## What Changed - `scripts/public-install-registry.mjs`: the 302 redirect for packages that the registry does not serve now uses npm's own escaping (`name.replace('/', '%2f')`, the same as npm-package-arg `escapedName`). The `packageName` check already allows only characters that need no other escaping. - `scripts/__tests__/public-install-registry.test.mjs`: the test now expects `@other%2fpkg` for a scoped redirect. A new assertion covers an unscoped redirect. ## Verification - `node --test scripts/__tests__/public-install-registry.test.mjs`: 3 of 3 pass. This includes the real npm install test that compares installed bytes from file installs and registry installs. - Local replay (Linux, npm 11.12.1). I filled a fresh cache with the warm-up script, then installed the same 62-dependency set (627 packages) through the registry: - Before: 291 cache misses, 28.3s. - After: 0 cache misses, 19.8s. - I also ran the real `scripts/public-install-registry.mjs` with a packed root package that depends on the full public dependency set. Result: 2 misses (only the loopback package and its tarball, as expected) and 1438 cache hits. - To compare on CI, look at `consumer_install_ms` in the `paperclip.public-npm-install.timing.v1` line of the Build log. Baseline: 68822 ms (run 39254079691) and 75175 ms (run 39251304886). ## Risks - Low risk. The redirect target is the same npmjs document. Only the URL escaping changes. npm still checks every tarball against the integrity in the packument. - If the npm cache holds stale scoped metadata, npm revalidates it the same way as it does for direct requests (`cache-control: max-age=300`). - The expected time saving comes from a local measurement. The real Build saving depends on the network and CPU of the runner. I expect about 10–20s per Build run. The CI timing line will confirm the number. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code as a Paperclip agent with tool use (shell, file edit). I did the root-cause analysis, the local replays and the code change with this model. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com> Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
ff7e7b4963 |
fix(claude-local): make sandbox tool commands the first OOM victims (#16844)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Hosted `claude_local` runs execute in a sandbox. The sandbox has a fixed memory limit (4 GiB in the observed sandboxes). > - In that sandbox, every process has the same `oom_score_adj`. When a Bash tool command uses most of the memory, the kernel kills the largest process. That can be the Claude Code process. > - When Claude Code is killed, `claude-agent-acp` reports "The connection to Claude was lost." and the whole run fails. Server logs show `Claude Agent process died: Claude Code process terminated by signal SIGKILL` for each failure. > - A failed tool command is recoverable. A dead agent is not. > - This pull request makes Claude's tool commands the first out-of-memory victims in managed sandboxes. > - The benefit is that a memory-heavy command fails with exit 137, and the agent continues the run. ## Linked Issues or Issue Description Refs #16366 (diagnostic PR for the same failure). Related: #14509 protects the host control plane from local agent processes. That PR does not cover a sandboxed agent and its own tool commands. **What happened?** Hosted `claude_local` runs fail with `acpx_turn_failed` and "The connection to Claude was lost." while one `execute` (Bash) tool call is pending. The pending commands are memory-heavy, for example `tsc --noEmit` with a 3.3 GB heap, a monorepo `pnpm install`, or a type check plus `vitest`. At the same second, the server log shows `Claude Agent process died: Claude Code process terminated by signal SIGKILL`. Over 10 days, all 12 such lines on one staging stack were `SIGKILL`, and they match the failed runs. The sandbox memory cgroup has `memory.max` 4 GiB and `memory.oom.group` 0, so the kernel kills one process. Every process has `oom_score_adj` 0, so the kernel picks the largest one. **Expected behavior** Under memory pressure, the kernel kills the tool command. Claude gets a failed command (exit 137) and continues the run. **Steps to reproduce** 1. Start a `claude_local` run on a sandbox target with a 4 GiB memory limit. 2. Ask the agent to run a memory-heavy command in a large repository, for example a full `tsc --noEmit` and a test run together. 3. The kernel kills the Claude Code process, and the run fails with "The connection to Claude was lost." **Agent adapter(s) involved** Claude Code (`claude_local`, ACP lane and CLI lane, managed sandbox targets). ## What Changed - `claude-config.ts`: add `CLAUDE_TOOL_SHELL_PREFIX_SCRIPT`. Claude Code calls a `CLAUDE_CODE_SHELL_PREFIX` with one argument, the full command string. The script raises its own `oom_score_adj` to 1000 (no privilege is necessary to raise it). Then it runs the command string in the shell that Claude Code selects: `CLAUDE_CODE_SHELL`, then `SHELL`, when either is an executable bash or zsh, else zsh or bash from `PATH` (bash first when `SHELL` names bash). - `buildRemoteClaudeConfigMaterializationCommand` can install the script (umask 077, write to `.partial`, `chmod 700`, `mv`). The install runs last in the existing materialization command, so it adds no sandbox round trip. It prints a marker only when the file is in place. - `materializeRemoteClaudeConfig` returns `toolShellPrefixInstalled`. It is true only when the command exits 0 and prints the marker. - ACP lane (`acp.ts`) and CLI lane (`execute.ts`): set `CLAUDE_CODE_SHELL_PREFIX` to the installed script only when the install is confirmed. A prefix path that does not exist would fail every Bash tool call, so a failed install keeps the previous behavior. An operator `CLAUDE_CODE_SHELL_PREFIX` in the adapter env is kept, and no script is installed. - `docs/adapters/claude-local.md`: add "Sandbox memory limits". ## Verification - `npx vitest run src/server/claude-config.test.ts` in `packages/adapters/claude-local`: 10 pass, 6 new. The new tests install the script with the real materialization command through `/bin/sh` and check: - the file content, mode 0700, the marker, and that no `.partial` file remains; - the command runs with `oom_score_adj` 1000 while the caller keeps its own score (Linux only); - shell selection: a valid bash `SHELL`, an invalid `CLAUDE_CODE_SHELL` that falls back to `SHELL`, and a `SHELL` of `/bin/sh` that falls back to bash from `PATH`; - quoting and the exit status of the command are kept; - an operator prefix wins, and no marker means no install. - `acp.test.ts`: the managed-config test now checks that `CLAUDE_CODE_SHELL_PREFIX` names an executable 0700 file next to the config dir. A new test checks that an operator prefix is kept and no script is installed. Both use the local sandbox runner, which runs the setup commands for real. - `tsc --noEmit -p packages/adapters/claude-local` is clean. - Two tests fail in my sandbox both with and without this change: `uses the same default model in ACP startup and session identity` and `reselects the full assignment and bootstrap guidance after a failed resume`. They also fail on clean `master` in the same environment, so I think they depend on that environment. CI covers them. - Real Claude Code check (Claude Code 2.1.286): with `CLAUDE_CODE_SHELL_PREFIX` set to the script, a Bash tool call printed `oom_score_adj` 1000 and ran in zsh (the selected shell), and its parent was the `claude` process at score 0. Without a prefix file that exists, every Bash tool call fails with exit 127. This is why the env is set only after a confirmed install. - Real OOM check in a 4 GiB Daytona sandbox (`memory.oom.group` 0). A stand-in "agent" process held 1.2 GB at score 0, and a tool command started 14 processes of 250 MB each: - without the prefix, the kernel killed the stand-in agent (SIGKILL, 137; `oom_kill` +1); - through the installed prefix, the stand-in agent stayed alive, and the kernel killed 5 tool processes (`oom_kill` +5). - Assumption that could invalidate the design: Claude Code passes one argument (the command string) to the prefix and honors the variable through `claude-agent-acp`. I checked the argument shape with a logging prefix, and `claude-agent-acp` 0.73.0 spawns the CLI with `{ ...process.env }`. ## Risks - Every Bash tool command and hook in a managed sandbox now runs through one more `sh` process and an `exec`. The cost is a few milliseconds for each command. - If a future Claude Code release changes how it calls `CLAUDE_CODE_SHELL_PREFIX`, Bash tool calls can fail. The tests above and the documented variable limit this risk. Removing the variable restores the old behavior. - The inner shell is not a login shell. Claude Code sources its own shell snapshot inside the command string, so the environment stays the same. If Claude Code runs without a snapshot, it uses a login shell. Only exported variables then pass through to the inner shell. - If the selected shell differs from the one that Claude Code chose (for example, a custom `CLAUDE_CODE_SHELL` that is not bash or zsh), the script uses the same fallback order as Claude Code. - This does not help when the Claude Code process itself uses most of the memory. The local (non-sandbox) lanes do not change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with tool use and code execution. It read the repository and the Claude Code CLI behavior, ran the OOM experiment, edited the code, and ran the tests. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass (two tests fail on clean `master` in my sandbox too; see Verification) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Nicky LeachandClaude Opus 5.5
|
e107a75755 |
fix(native): decide session settlement from live state (#18992)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native Runner sessions keep one provider conversation across task turns, so a later run attaches to a session that an earlier run settled. > - Three gates decide if a session is settled: Core's journal check, the Runner's attach check, and the Runner's drain of provider messages after a turn. > - Each gate kept a list of harmless events and refused anything else. Every new harmless event became an outage. > - An example: Codex replays a settled turn's usage on `thread/resume`, so `run.attach` refused every session that restarted after its idle timeout. > - Events after a terminal describe the past. Refusing the session cannot undo them, and it hides them. > - This pull request asks a different question: is anything live or waiting right now? Past events are recorded and no longer block. > - The benefit is that harmless events stop breaking session continuity, with no event list for Core and the Runner to keep in step. ## Linked Issues or Issue Description No issue exists. I searched open and closed pull requests and issues for `run.attach requires a settled Codex provider session` and `prior_owner_terminal_state_indeterminate`. Related work: #18890 added one more accepted event to Core's list. This change replaces that list. #18991 handles late usage and is independent of this change. **What happened?** On a test deployment, 11 runs failed in 21 days when `run.attach` refused with `requires a settled Codex provider session with no pending events`. In all 11, the four pending events were two startup audit events, `session.resumed`, and a `codex_resume_usage_snapshot` diagnostic. The diagnostic is what the attach check refused. The retry then failed with `prior_owner_terminal_state_indeterminate` in 6 of them, which hid the first failure. **Expected behavior** A session with nothing live or waiting accepts the next run. If attachment fails, the run reports the real cause. **Steps to reproduce** 1. Finish a Codex turn on a native Runner session. 2. Wait for the idle timeout so that the provider process stops. 3. Start the next run. Codex replays the usage for the settled turn on `thread/resume`. 4. The attach fails. `durable_backend_attaches_after_a_settled_resume_usage_snapshot` reproduces this on the unfixed code. **Paperclip version or commit** Observed on `8fd917aedf`. The same check is on `master` before this change. **Deployment mode** Cloud deployment, with a native Runner in a plugin sandbox environment. ## What Changed - **Runner attach:** `attach_run` refuses only for live or waiting state: no thread, a closed session, a session that needs reconciliation, an active or ambiguous turn, an active goal, a pending tool call, or an unanswered provider request. The error now names the reason. - **Pending events:** they no longer block attachment. They stay in the retained queue, and the Runner commits them under the prior authority before it rotates. `session.resumed` is still discarded because it is superseded. - **Runner drain:** `drain_completed_turn_tail_for_warm_attachment` no longer needs a list of known messages. It still fails on a message that names another turn, on `turn/started`, on a provider request, and on a process exit. - **Core:** `isSettledRemoteWarmControlState` accepts any authenticated event after the terminal unless it starts work (`turn.started`, `turn.accepted`, `run.attached`, `tool.execution.started`, `semantic_tool.input`, `runtime_request.created`), or a goal snapshot or update shows an active goal or work in progress. All identity, source and protocol checks are unchanged. - **Visibility:** the next run's log names the event types that were recorded after the previous run finished. - **Real cause:** a same-run retry after a failed attachment now fails with `prior_attach_failed: <original failure>`, instead of `prior_owner_terminal_state_indeterminate`. - **Docs:** added "Session settlement after a run" to `doc/execution-semantics.md`. ## Verification - `cd packages/paperclip-runner/runner && cargo test --locked -p paperclip-runner-core --test codex_provider`: 102 passed, 2 ignored. `--lib`: 478 passed, 2 ignored. `--test durable_recovery`: 3 passed. - **Failure before the fix:** `durable_backend_attaches_after_a_settled_resume_usage_snapshot` failed on the unfixed code with the same error as production. - `cd server && pnpm exec vitest run src/services/native-runtime/remote-warm-session-recovery.test.ts src/services/native-runtime/native-session-executor.test.ts -t "settled remote warm control evidence|recovers a blocked|automatically recovers only a proven settled local session|retained|warm"`: 476 passed, 680 skipped by the filter. - `pnpm exec vitest run src/services/native-runtime/native-session-resume.test.ts`: 57 passed. - `npx tsc --noEmit -p .` in `server` reports no errors. - **Real data, read-only:** I decrypted the 11 failed runs' stored events on a test deployment and printed only event types, field names, and thread and turn IDs. All 11 show the same four events. - **Audit of live state:** I read the Runner's provider state and request tracking to build the attach check. It is a code audit. I have not proved that every kind of live work sets a flag. The Runner holds active turns, ambiguous starts, tool calls and provider requests in state today. - **Not covered:** the wiring that throws `prior_attach_failed` has unit coverage for its message only, not an end-to-end test. - **Review:** Greptile Apex scored 4/5, then 5/5 at the current head. - **Not run locally:** the full test suite. CI provides that coverage. - **Not verified live:** no live run has used this change. ## Risks - **Fail-open for unknown live work:** an event type that starts work, but is not in the short list and sets no state flag, would now pass. This is the main tradeoff. The previous list failed closed, but it caused real outages. - **More events reach the journal after a terminal.** The usage replay diagnostic is now delivered rather than refused. These events are informational, and they are named in the next run's log. - **A failed attachment is not retried on the same run.** Some refusals could clear on their own, such as a turn that finishes, and some attach errors are not about live work, such as a failed provider restore. The run now reports the cause, and it needs another run to continue. On a test deployment, the next successful run after each failure used a new provider thread and a new execution workspace. A later change could add same-run retry. - **Rollout order:** deploy Core before the Runner. An older Core would refuse the new events that a new Runner commits after a terminal. - **Migrations:** this change has no migration. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be reworked. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), through Claude Code, with tool use and code execution. A human directed the design and approved the scope. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
25b5a072cc |
fix(recovery): let messages continue recovered preparation (#19026)
## Thinking Path > - Paperclip must preserve work while allowing people to continue failed tasks. > - A preparation failure can retain an ID from an earlier provider session. > - Workspace recovery and host settlement already prove that the failed attempt never dispatched a provider. > - The preparation classifier also required that inherited session ID to be absent. > - That extra condition kept new messages, saved messages, and Retry behind a resolved recovery hold. > - Classify execution from the existing execution evidence and admit one fresh turn through normal gates. ## Linked Issues or Issue Description Refs #18908, #19004, and #19021. The latter two address historical Retry ancestry and empty object logs. This repair addresses a current source with an inherited session and ordinary user messages; it needs neither ancestry nor an object log. ROADMAP.md and related open PRs were checked. **What happened?** Sending a message to a failed task saved it in the queue, but did not start another run. The failed native preparation had recovered files and durable host settlement, plus `sessionIdBefore` from prior work. The classifier treated that input as execution evidence and kept the recovery hold. **Expected behavior** Admit the new user turn once the existing saved-work and stop checks pass. Preserve previous failures and files. Show the current run instead of the old error, and consume messages when execution starts. **Steps to reproduce** Fail Runner preparation before runtime selection. Preserve its remote workspace. Retain an inherited `sessionIdBefore`. Send an ordinary message or click Retry. The three new HTTP regressions accept the request but admit zero successors before the repair. **Paperclip version or commit** Base `3bd52a9fa0`. **Deployment mode** Authenticated hosted Runner. Regression verification uses macOS arm64, Node 26.4.0, real PostgreSQL, HTTP routes, and real temporary workspace archives. The external sandbox transport and native provider boundary are controlled in the execution fixture. ## What Changed - Stop requiring the inherited session input to be absent from otherwise verified failed preparation. - Cover new messages, previously saved messages, Retry, request replay, one successor, recovered files, and current task presentation. - Exercise the existing preparation safety controls with inherited context, including a new output-session refusal. - Exercise native execution and finalization with the inherited session input. - Document the continuation rule. ## Verification - RED: all three new HTTP cases admitted zero successors before the production change. - All three final HTTP cases pass. The 41 preparation safety cases, including the new output-session refusal, pass with inherited context. - The native execution fixture passes with inherited context. It reads restored files, writes the continuation result, runs one provider turn, persists the accepted result, empties the queue, and resolves the notice. - Initial positive assertions expected an empty queue while the fixture deliberately held the agent's slot, then expected a null optional wait field. The final assertions use the actual queue contract and retain execution and presentation checks. - `pnpm build`, `node scripts/typecheck-low-memory.mjs`, module boundaries, and `git diff --check` pass locally. The final rebased head `56d2ef26` passes the four HTTP/native journey cases. All 634 tests in the four complete owning recovery, Retry and continuation suites pass. Current-head full CI passes: 53 successful checks and three expected skips, with no failed or pending checks. The full local command has the documented timeout below. - The full local command passed its first three chat shards, then stopped in shard four on a 15-second timeout in `answers each finding thread in its own run instead of one unattestable batch` (267 other shard cases passed). The exact unchanged test passes alone (about nine seconds of test work) and all three full CI chat groups pass on this head. It uses a custom adapter, so the changed native-preparation proof is inapplicable. The timeout cause remains unknown; the full local command is not claimed green and its later groups were not reached. - Initial CI attempt: all other checks passed, but server shard 2/12 failed on the unchanged `idle-completed-history.test.ts` owned HTTP runtime test. The ten-second readiness probe returned ECONNREFUSED. The source and runtime have no diff from the base and this test does not use failed-preparation admission. The exact local case passes; the underlying readiness cause remains unknown. Logs and annotations are retained at [the failed job](https://github.com/paperclipai/paperclip/actions/runs/41897539206/job/126811385017). The single failed-job retry passed on the same head, including the dependent verify aggregate. This proves recovery of that check; the original readiness cause remains unknown. - Greptile reviewed the exact head `56d2ef26` at 5/5 with no actionable findings and zero unresolved threads. Hosted recovery remains unverified until the repair reaches the serving revision. ## Risks The risk is mistaking earlier context for a stopped current execution. Current provider session, process, controller, native input/result/output, accounting, exact lease stop, workspace epoch/generation/writer, actor, assignment, pause and approval checks remain enforced. No migration, new polling, API, credentials or execution policy is added. ## Model Used OpenAI GPT-6 through Codex, using reasoning, code editing, shell tools, and browser diagnostics. The exact serving snapshot and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (634 owning-suite cases and final-head journey cases; full-local timeout is documented above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
e65c68726b |
Require published migrators before publishing Core images (#19023)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Its release pipeline publishes Core images and exact-source database
migrators.
> - A migrator run can pass admission but skip all build and publication
work.
> - The image observer could stop checking after provisional build
admission.
> - The final image gate checked source proof without checking migrator
publication.
> - This pull request requires the exact migrator publisher to finish
before image publication.
> - Consumers can then use the published image with its matching
migration artifact.
## Linked Issues or Issue Description
**What happened?**
The [migrator run for
|
||
|
Devin FoleyandClaude Opus 5.5
|
09f9d691cf |
fix(server): keep the root failure code when the execution deadline fails a run (#19028)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The execution-control watchdog fails a run when its control deadline passes and no owner finished it. > - The watchdog replaces the run's error code with `execution_finalization_deadline_exceeded`. > - The run-failure report reads the root cause from `resultJson.originalFailureCode`. The watchdog did not write that field. > - So every watchdog failure goes to Sentry with the same fingerprint, whatever its first failure was. > - This pull request keeps the earlier failure code as `originalFailureCode`, as the native finalizer already does. > - The benefit is that Sentry groups watchdog failures by their real first failure, so each cause can be found and fixed. ## Linked Issues or Issue Description **What happened?** One hosted Sentry group collects all `execution_finalization_deadline_exceeded` run failures. It has 17 events from many drivers (Codex app server, OpenCode server, ACPX runtime). In one recent event, the run first failed with `native_execution_ownership_unverified`. The watchdog then failed it at the deadline. The recovery action evidence kept `originalFailureCode: native_execution_ownership_unverified`, but the run row and the Sentry event did not. The event had no `root_error_code` tag, and its fingerprint was only `[execution_finalization_deadline_exceeded, paperclip_runner]`. **Expected behavior** When the watchdog replaces an earlier, more specific error code, the run keeps that code as `resultJson.originalFailureCode`. The run-failure report then sends `root_error_code` and adds it to the fingerprint (`server/src/sentry.ts`). **Steps to reproduce** 1. Create a running run with `errorCode: native_execution_ownership_unverified` and an expired `executionControlDeadlineAt`. 2. Run `reconcileAbandonedExecutionControl`. 3. The run fails with `execution_finalization_deadline_exceeded`, and the earlier code is lost. The captured event has no `diagnostics.execution.originalFailureCode`. **Paperclip version or commit** `master` at `89703c3ff7de`. Refs #14110, a similar fix for the finalization sweep. A search found no other open PR for this path. Draft #18672 changes legacy deferred wakes, which is a different path. ## What Changed - `execution-control-reconciliation.ts`: when the watchdog fails a run, it writes the earlier code (`coordinator.failureCode ?? run.errorCode`) to `resultJson.originalFailureCode`. This is the same value that it already writes to the recovery action evidence. - The watchdog keeps the other `resultJson` fields. - The watchdog does not replace a root cause that is already recorded. - The watchdog does not record the deadline code as its own root cause. - Two regression tests in `execution-control-reconciliation.test.ts`. ## Verification - The new test "keeps the run's prior failure code as the reported root cause" fails on unchanged `master`. The run's `resultJson` has no `originalFailureCode`. With the fix, the test passes. - `pnpm exec vitest run src/services/execution-control-reconciliation.test.ts src/__tests__/execution-control-work.test.ts src/__tests__/run-failure-sentry.test.ts src/__tests__/sentry.test.ts`: 111 passed, 5 skipped. - `pnpm exec vitest run src/services/task-retry.test.ts src/services/heartbeat/recovery.test.ts src/services/native-runtime/native-safe-replacement.test.ts src/services/workspace-recovery.test.ts`: 437 passed. These suites use this watchdog cause. - `node scripts/typecheck-layers.mjs` in `server/`: passed. - Not run locally: the full `pnpm -r typecheck` (the sandbox has no Rust toolchain for the runner crate), `pnpm test:run` and `pnpm build`. CI runs these. ## Risks - Low risk. The change adds one `resultJson` key only when the watchdog moves a run to `failed`, and only when a different earlier code exists. - Other readers of `resultJson.originalFailureCode` (`native-run-projection.ts`, `idle-sleep-safety.ts`) only check committed or succeeded runs. A watchdog failure is neither, so their results do not change. - `isPendingNativeRetry` reads only `nativePhase`, so reporting is not suppressed. - Sentry groups will split by root cause after deployment. New events will go to new groups, and the existing group will get fewer events. This is the intended change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution in a Paperclip agent sandbox. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
3a283fbc9b |
fix(runtime): preserve failed sources with older warm supervisors (#19024)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A volume-free web server needs durable recovery data for retained sessions. > - An old failed follow-up can retain its snapshot while the warm supervisor still holds the preceding successful turn. > - The migration refuses to treat those different snapshots as the same execution authority. > - That refusal also blocks the tenant even when the failed source can be preserved as an archive. > - This pull request archives that narrow case after it verifies the older successful snapshot and all scope bindings. > - The failed run keeps its history and files while valid controller recovery keeps its existing proof requirements. ## Linked Issues or Issue Description Refs #18959, which added the existing failed-controller archive path. No duplicate PR for this case was found. **What happened?** Offline controller migration stops with `stale_supervisor_snapshot` for an old failed warm follow-up. Its local supervisor still contains the prior successful turn. Read-only checks found the same database snapshot and supervisor bytes in the live source and the isolated copy. The mismatch predates the rehearsal. **Expected behavior** Preserve the failed run's database state and available local files in the existing durable archive. Only archive when the older supervisor matches a completed successful run in the same company, agent, task, native session and Runner instance. Keep current execution and controller authority checks intact. **Steps to reproduce** Create a successful native turn and a later failed warm follow-up. Keep the successful snapshot in the supervisor file and the failed snapshot in the database. Give the failed source local controller bytes but no durable controller checkpoint. Run the offline migration. The new regression fails with `stale_supervisor_snapshot` before this change. ## What Changed - Verify the older supervisor against scoped successful history and the same provider thread. - Use the existing encrypted archive owner for that failed legacy source. - Preserve snapshots, run status, ownership, queued work and the session's last-run pointer. - Keep foreign, changed, active, successful and already-durable sources outside this path. - Document the preservation and recovery boundary. ## Verification - New positive regression: red on the original owner, green after the change. - Native migration and controller recovery: 27 checks passed. Final migration checks also cover refusal to replace a durable checkpoint. - Storage failure blocks publication. Repeated migration is safe. The archive restores the exact controller and supervisor bytes after the old home is deleted. - The archived failed source does not gain executable recovery authority. - Local server typecheck is blocked by reused workspace declarations for accounting, bootstrap and ingress contracts. No diagnostic names either modified owner or test file. Full local typecheck, repo-wide tests and build are not claimed; clean CI is required before merge. The exact-head CI has now passed typecheck, build, policy, Runner checks, browser E2E, and all server test shards. - Rehearsal verification against the copied historical source remains required. Full tenant migration and hosted deployment qualification remain separate gates. ## Risks The archive preserves evidence for an already failed source. It does not certify that source as resumable. A prior snapshot must match exactly, and the normal archive owner checks the terminal failed binding before it publishes. Existing durable checkpoints remain authoritative. A missing or changed proof still blocks migration. ## Model Used OpenAI Codex, exact model ID `gpt-6.1-sol`, with reasoning, tool use and code execution. The host does not expose context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR following the bug template - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal ticket id or instance-derived details - [x] I have run the relevant tests locally; retained failures and limits are listed above - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>canary/v2026.1104.0-canary.12 |
||
|
Devin FoleyandPaperclip
|
cf0ab4f7f4 |
chore: enable boxd plugin release publishing (#19020)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Sandbox provider plugins let those agents run in remote environments. > - The boxd provider exists, but its release entry excludes it from CI publishing. > - npm now has the `@paperclipai/plugin-boxd@0.0.0` bootstrap placeholder. > - This pull request enables CI publishing and documents npm installation. > - A pre-existing OpenCode fixture blocked CI because it qualified a group-writable host Node binary. > - This also gives that test its own Linux Node copy with strict permissions. > - Operators can install a real release after CI publishes it. ## Linked Issues or Issue Description Related implementation: https://github.com/paperclipai/paperclip/pull/17284. No matching public issue or duplicate release-enrollment PR was found. **What existing behavior does this improve?** Publishing and installing the existing boxd sandbox provider. **Subsystem affected** `packages/plugins` and the npm release manifest. **Current behavior** `publishFromCi` is false. npm has only the `0.0.0` placeholder. The README says the package is not published and requires a local-path install. **Proposed behavior** Include boxd in CI releases. Document stable installation, explicit published versions, and the placeholder limit. **Reason and benefit** Operators can install the provider through the existing npm plugin installer after a real release ships. **Breaking changes** None. This changes release enrollment and documentation only. **Additional context** This follows the existing plugin and sandbox roadmap items. It adds no new runtime capability. ## What Changed - Set `publishFromCi: true` for `@paperclipai/plugin-boxd` in the release manifest. - Replace the README's unpublished-package notice with npm installation guidance. - Explain that the `0.0.0` placeholder has no plugin functionality and that `latest` can point to it until the first stable release. - Keep the local development instructions available. - Fix the existing ingress-retry test to use an owned Linux Node copy and explicit executable permissions. This follows the existing OpenCode transport fixture. It does not change production launch checks. ## Verification - Passed: `node --test scripts/release-package-map.test.mjs scripts/check-release-package-bootstrap.test.mjs scripts/bootstrap-npm-package.test.mjs` (31 tests). - Passed: `node scripts/release-package-map.mjs check`. - Passed: `PAPERCLIP_RELEASE_BOOTSTRAP_BASE_SHA=origin/master node scripts/check-release-package-bootstrap.mjs scripts/release-package-manifest.json packages/plugins/sandbox-providers/boxd/README.md`. - Passed: `git diff --check`. - Confirmed that npm resolves `@paperclipai/plugin-boxd` to version `0.0.0`. - Full workspace typecheck, test suite, and build were not run locally. CI checks remain required before merge. - Initial CI failure: [native Runner job](https://github.com/paperclipai/paperclip/actions/runs/41889969053/job/126785636239), OpenCode ingress retry, `qualified OpenCode proxy command must not be group- or world-writable`. The failing fixture is byte-identical to the base commit `3a2249dd576`. It passed `process.execPath` to qualification, while other OpenCode tests already use a private Linux Node copy. - Local focused run: `pnpm exec vitest run server/src/services/native-runtime/native-ingress-bootstrap-retry.integration.test.ts` built the real Rust Runner and passed the Codex case. The OpenCode case stopped at `runner_remote_provider_artifact_incompatible: OpenCode proxy path belongs to the controller host` on macOS (`/Users/...`). This does not exercise the new Linux-only copy. - An unchanged-base OpenCode comparison was attempted with `-t "reacquires opencode ingress"`, but PostgreSQL setup failed with `ENOSPC` before the case ran. The local baseline comparison is therefore inconclusive. Task-owned Rust build outputs were cleaned to recover disk space. - At this stage Linux verification was pending. The successful final run is recorded below. - No new tests were needed. The existing release-tool tests cover manifest enrollment, dependency ordering, and bootstrap checks. - Current-head CI interruption: [workspaces-a shard 1/2](https://github.com/paperclipai/paperclip/actions/runs/41893808561/job/126799012853) was cancelled after the runner received a shutdown signal. The log shows no assertion failure. The job rerun request was rejected with HTTP 403 because the workflow was still running. The subsequent fixture correction starts a new full CI run and supersedes that pending rerun. - CI at `7fffa315`: [native Runner job](https://github.com/paperclipai/paperclip/actions/runs/41893808561/job/126799011711) passed executable permission checks, then failed because Node tried to resolve `/memfd:paperclip-verified-launch (deleted)`. The private executable was named `opencode-node`, which bypassed Runner's Node interpreter recognition. The fixture now names its copy `node`, matching the existing transport fixture and selecting the CommonJS descriptor loader. - Passed a local smoke test of the production CommonJS descriptor loader using an open, unlinked script file. This is macOS evidence only; the successful final Linux run below provides the complete ingress retry evidence. - Final verification on `bcacac542aa0a1b641b26c099c1e9fc4a1653ec4`: all 54 reported checks passed; three unrelated checks were skipped. [Native Runner integration job](https://github.com/paperclipai/paperclip/actions/runs/41895787712/job/126805844773) passed, including the OpenCode ingress retry regression. The fresh full run also passed the workspace shard that was interrupted on the previous attempt. Greptile reports 5/5 for this exact head with no unresolved review threads. Earlier failures and local verification limits remain recorded above. ## Risks - The maintainer confirmed that npm trusted publishing is configured for this package. The first real CI publish will verify OIDC authentication. - Merging enables publication in the next successful canary run. An installation that uses `latest` can still resolve to the placeholder until the first stable release includes this change. ## Model Used OpenAI Codex, exact model ID `gpt-6.1-sol`, with xhigh reasoning effort, repository tool use, and shell execution. The runtime does not expose the context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Dotta
|
89703c3ff7 |
fix(ui): place pinned sidebar pages above More (#18915)
## Thinking Path > - Paperclip helps people manage AI agents and their work. > - The Coordinator sidebar gives people access to workspace pages. > - People can pin pages from the More menu. > - The sidebar puts these links below More in a separate Pinned section. > - This change puts pinned links above More in the main navigation. > - People can reach their saved pages without an extra section heading. ## Linked Issues or Issue Description **What existing behavior does this improve?** The placement of pinned pages in the Coordinator sidebar. **Subsystem affected** ui/ — React board UI. **Current behavior** Pinned pages appear below More with a Pinned heading and extra space. **Proposed behavior** Pinned pages appear after Needs you and before More. They use the same row spacing as the main links. **Reason and benefit** Saved pages belong with the main navigation. This placement keeps them visible above the menu that contains the other pages. **Breaking changes** The Pinned section and its heading are removed. Saved pin data, link order, routes, and pin controls use the existing code. ## What Changed - Move pinned links into the primary navigation above More. - Remove the separate section, heading, and extra space. - Correct the feature guide to state that pinned pages appear above More. - Update sidebar tests to check placement, saved order, account changes, restore, and pin removal. ## Verification - `pnpm exec vitest run ui/src/components/CoordinatorSidebar.test.tsx`: 15 tests pass. - `pnpm check:token-gates`: pass. - `node scripts/check-maps.mjs`: pass after the guide correction. - `pnpm --filter @paperclipai/ui typecheck`: pass. - Browser checks pass in Chromium against the production Coordinator sidebar in Storybook with fixture data. Screenshots cover saved pins and pinning from More. No page errors occurred. This does not test a live API. - The full workspace typecheck and build stop because this host has no Rust `cargo` command. - The first UI build was killed and the browser crashed while checks ran together. The host recorded OOM kills at its 8 GB limit. The browser check then passed after the full-suite process was stopped. The UI production build passed after the Storybook server was stopped. - `NODE_OPTIONS=--max-old-space-size=4096 pnpm --filter @paperclipai/ui build`: pass. - `pnpm test:run`: stopped during the first chat integration shard after memory exhaustion affected other checks. It has no full-suite result. The CI run passed the test lanes on the latest head. - [CI run](https://github.com/paperclipai/paperclip/actions/runs/41768881396) passed on `a836fc1320c363e0fbb5591312c6726026b6f151`. The required `ci / verify` and `ci / e2e` checks pass. Build, typecheck, Rust checks, test lanes, and all eight browser shards pass. - Greptile: 5/5 on this head. The documentation finding is fixed and its thread is resolved. - The additional Paperclip review is paused and queued. It is not a required branch check. ## Risks Low risk. This change moves existing links. It does not change storage or API behavior. The tests check expanded and collapsed rendering, the empty state, stored order, unknown routes, user changes, and changes from another browser tab. ## Model Used OpenAI GPT-6 through the Codex agent. The session supports reasoning, tool use, code edits, and command execution. The runtime does not expose a more specific model ID or context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge |
||
|
DottaandPaperclip
|
d180ad8c77 |
fix(heartbeat): retry chat execution authorization contention (#19017)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A task Retry must use the current task and chat authority. > - The request and queue paths already retain a Retry through brief lock contention. > - Execution checks that authority twice, including after native input preparation. > - A busy task row or chat identity could fail the admitted run at either check. > - This pull request retries only the rolled-back authorization transaction. > - The same run can proceed after the lock clears. The final dispatch gate still rejects changed ownership. ## Linked Issues or Issue Description **What happened?** A failed chat task can accept a normal Retry and then fail during execution setup when another PostgreSQL transaction holds the task row or chat identity lock. The late check can fail after native input and workspace preparation. The exception can then enter the workspace preservation path. **Expected behavior** Keep the same admitted run and its frozen input. Retry a recognized, rolled-back authorization conflict within the existing bound. Recheck current authority before each attempt. Reject Stop, paused agents, revoked chat access, or a changed task owner before the execution callback. **Steps to reproduce** 1. Admit a normal failed-chat task Retry with the existing PostgreSQL fixture. 2. On a second connection, hold the task row or the chat identity advisory lock when execution authorization starts. 3. Release the lock after the real authorizer reports contention. 4. Observe that the original code fails the run before the authorized callback. Repeat at the late native check after input preparation. **Paperclip version or commit** Reproduced on master `039e0dad2c51c80251e1302a19c5566e039c357a`. The branch is rebased on `8f06bd4e1`. **Deployment mode** Source checkout with real PostgreSQL and the local HTTP Retry route. Provider and chat transport responses use controlled test fixtures. Refs: #18895 and #18896 cover request and queue admission. This change covers execution authorization. #18898 and #18989 cover predecessor cleanup, a separate boundary. The roadmap and public PR search show no duplicate execution-authorization repair. ## What Changed - Recognize the existing typed chat-presentation contention error in the bounded admission retry helper. Keep its existing PostgreSQL lock-unavailable handling. - Use that helper at both execution authorization checks. Each attempt checks cancellation and the applicable legacy controller lease, then opens a fresh transaction. The loop contains no provider or workspace callback. - Include failed-chat Retry in the existing final dispatch ownership gate. This preserves current task ownership and native recovery owner checks after a delay. - Add real PostgreSQL contention, late prepared-input, Stop, pause, revocation, owner replacement, native completion, and ordinary legacy completion controls. Document the retry and dispatch boundaries. ## Verification - The unchanged base failed real task-row and chat-identity contention cases. A separate late native case failed after frozen input preparation. A safety control then proved that a replaced task owner could still reach the backend; the final dispatch gate fixes that path. - All 11 PostgreSQL cases and all four helper controls pass. The tests keep exact frozen model and trust input, reject changed authority, and do not replay a provider callback. - A real local HTTP Retry reaches the native executor, one controlled provider turn, one accepted result, a committed finalizer, and one published chat reply. Reprocessing the canonical request does not start another turn. - A separate real HTTP ordinary legacy Retry runs one controlled adapter callback, completes its task through the normal service, and does not replay the canonical request. - Independent review found no material issue in the two production files. It checked transaction scope, lock order, current ownership, native lease expiry, and import cycles. - The full workspace typecheck and full `pnpm build` pass on this head. Full `pnpm test:run` passed all 1,916 chat integration cases. Its broader server group reported the existing workspace-runtime case `returns a bounded structured conflict and exposes only same-company workspace references` as failed after 46 ms. The unchanged focused case and the complete 179-case file both pass on rerun. The active broad run was then stopped for this finite handoff; its child exited 130 and the command exited 1. The detailed assertion was buffered and was not retained, so its exact cause remains unknown. This is a partial local run, not a full local pass; later local groups did not run. All three CI chat shards pass. [Greptile reviewed this exact head at 5/5 with no findings](https://github.com/paperclipai/paperclip/pull/19017#issuecomment-6638521119). - Retained setup and fixture failures are not counted as product regressions or passes. An incomplete copied Copilot runtime failed the first typecheck; the supported installer repaired that isolated dependency, then typecheck passed. Early acceptance fixtures used a wrong publication field or did not complete their task; the final controls use the real field and completion service. - The first CI run failed [workspace shard 1](https://github.com/paperclipai/paperclip/actions/runs/41885475363/job/126769239992) on child-server startup and a five-second dense ZIP test deadline. [Server shard 2](https://github.com/paperclipai/paperclip/actions/runs/41885475363/job/126769239908) failed an existing local runtime readiness probe. All 18 cases in the two CLI files and the runtime case then passed unchanged locally on this head. The CLI files also match an earlier independently verified head byte for byte. One failed-job rerun was requested. Its [server job](https://github.com/paperclipai/paperclip/actions/runs/41885475363/job/126777070120) was cancelled after the runner received a shutdown signal. It produced no test verdict. After that workflow ended, only the interrupted server job and its required dependents were rescheduled once. The [complete workspace rerun passed](https://github.com/paperclipai/paperclip/actions/runs/41885475363/job/126777070125). The [rescheduled server job](https://github.com/paperclipai/paperclip/actions/runs/41885475363/job/126781960353) and its aggregate passed. The [complete current-head workflow is green](https://github.com/paperclipai/paperclip/actions/runs/41885475363). All required current-head CI and security gates pass, including the Ready-triggered scans; three conditional checks are skipped. The complete unchanged idle-history file also passes all 75 cases locally. No timeout or assertion changed; all attempts remain part of this result. - These local controls do not claim hosted-provider or Linux sandbox acceptance. The original task-row failure did not retain a SQLSTATE; the PostgreSQL reproduction proves the local gap without assigning that original cause. ## Risks - The existing bound allows 50 backoffs of 100 ms. It limits backoff time, not total database or lease wall time. - Native ownership is checked at the final dispatch gate. The per-attempt controller lease assertion applies to legacy execution. - A persistent lock or changed authority still fails closed. Unknown prior provider effects and accepted work retain their existing guards. - No migration or model, account, trust, credential, or permission policy change is included. ## Model Used OpenAI GPT-6 through Codex, with reasoning, tool use, code editing, and PostgreSQL verification. The host does not expose the exact serving model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass within the stated scope: relevant regression and acceptance controls, and the complete chat, idle-history, and workspace-runtime files. The broad local run remains partial as recorded above. - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
3bd52a9fa0 |
test(evals): pin task-history discovery and governed handoff repairs (#18995)
## Thinking Path > - Paperclip must complete work through ordinary human and agent controls. > - The paired timeline attempt omitted a real human handoff, although the other ten checks passed. > - The agent used the always-exposed active-task history tool as if its comments were a status audit. > - The tool now states its actual scope and directs chronology and handoff questions to the existing discovery and API tools. > - A separate approved-tool attempt preserved permission and write boundaries, but paired Markdown emphasis caused a false outcome rejection. > - The reviewed oracle normalizes that formatting before the existing outcome checks and retains balanced negatives. > - This PR pins those corrections with the already reviewed personal secure-handoff admission repair for one new representative batch. ## Linked Issues or Issue Description Refs #18962, #18968, #18971, #18972, #18977. Follows merged #18976. The original paired results remain FAILED. Result 4412 omitted the actual human handoff; this is a real answer gap, not a date-oracle excuse. Result 4411 correctly handled the first rejection with no writes or Trust action and rejected agent permission, but the observer misread paired inline emphasis in the truthful outcome. The personal-secret result 4378 remains FAILED at admission; secure save and worker proof were not reached. No prior grade is rewritten and no new native pass is claimed. ## What Changed - Pin immutable candidate `249b487a2d2451763913038cc009f7090580d1f1` on `codex/chat-catalog-human-answer-admission-candidate-20261104`. - Extend exact prior source `b1119630003785c9e26e14c763859125642ad756` with only the accepted timeline-discovery and formatting corrections: 12 source, test and generated files, with 72 additions and ten removals. - Use the canonical `get_task_history` description in the actual exposed semantic catalog. It reads only bounded active-task comments; calling `get_task` does not change that active scope. It is not a status or assignment audit. Progress and handoff questions must discover the company timeline or issue activity, then use the existing API call. - Regenerate semantic catalog and protocol projections on this frozen candidate. Preserve all non-description fields, schemas and authority boundaries. Do not copy newer master catalog or document-schema changes. - Normalize paired inline emphasis before approved-replay outcome checks. Declare only that oracle as app-review v6. Keep timeline v5, personal v5 and company v4. - Retain the reviewed personal completion-context, addressed-human, optional-note and rendered-placeholder safety controls. Actual private save, owner isolation, audit, authorized worker delivery and native continuation remain required. - Update only the existing manifest, expected pin and descriptor snapshot in this infrastructure PR. Preserve all 244 identities, original requests, run/human counts, clocks, model, launcher bytes, privacy limits and qualification holds. ## Verification - The combined candidate passed 542 existing owning controls across 19 suites, including catalog contracts, consistency and 242 lazy declarations with no fixture callbacks. The manifest retains two legacy tasks, for 244 identities. - Candidate Runner checks passed: 15 semantic consumer tests and 12 catalog tests. The owning Runner TypeScript compiler passed after the existing eval-kernel prerequisite built. The first compile reported missing ignored eval-kernel output; the repair used candidate-owned ignored dependency links and output only. No repository type or acceptance rule was changed. - All three generated checks passed: semantic-action catalog, semantic contracts and protocol manifest. Deep comparisons prove each generated catalog differs only in `get_task_history.description`. The seeded fixture changes only its catalog hash, and the protocol manifest changes only the hash of that fixture. The compiled canonical catalog hash matches the fixture. The candidate import conflict added only `getTaskHistoryAction`; no unrelated newer document-schema import was copied. - Both owning fixes cleared fresh exact-head Greptile 5/5 with no actionable threads before this publication: timeline `7f6ba9686b5edba005a49cd15c0d0b52248178dc` and approved replay `0015a79c1e0a3af5292edf4a61aa9bab8f381899`. The personal prerequisite `789bec74a0f43011723fff45596ef9d084bfaba7` also cleared exact-head 5/5 and all 12 threads before its earlier integration. - All 131 existing pin, TestOps contract/API, adapter, reconciliation and streaming controls passed. The descriptor snapshot matches the actual candidate bytes. The only new declaration change from the prior candidate is app-review v5 to v6. Whitespace checks passed. - The SDK source guard reports 506 unchanged inputs, runtime pins unchanged, registry `efa70621e9fb42aa185f7ee616a1a726aa3eccb367e67e28817cedf26ef2ade8`. No SDK registry or runtime certificate renewal occurred. The existing reporting-lock guard returned `already-reviewed` with lock `60f4c06bca47af3ab9d74281f2ef2e1ff559d566b862ba51c2fbe6c8e5bac35b`. - Historical CI results remain recorded. Head 73ea had a server-5 cancellation with exact AWS Spot-reclamation proof. Head 16134 had 44 successful jobs, three failed leaves and dependent verify failure, plus one selector skip. Two failed hosts had exact Spot termination proof; server 11 had two real deadline PostgreSQL failures. After an initial libzstd setup block, the unchanged four-test suite passed locally on exact merge `70e68bff5fb89251f769671ce9ab949ab5bf7fc9` in 33.69 seconds with its original deadlines and assertions. This does not establish the CI cause. One recovery POST returned HTTP403; zero replications were accepted. - Prior head c95 finished with 41 successful jobs, six failures, one cancellation and one selector skip. Four leaves had exact Spot-termination proof; three of their execution logs were unavailable. Server 11 instead had a real `live-recovery-work` next-deadline assertion failure, and server 4 had a real workspace-health connection-refused failure. Dependent verify failed. Windows passed. No replay or claimed test fix erased those results; their two assertion causes remain unresolved. - Exact head `1e89df928f7ccb43bf7af3b814f2c2207d75883a` now has successful normal CI run 41888521175 (attempt 2: 48 successful jobs and one selector skip) and successful Windows run 41888521015. The first attempt retained 46 successes, the failed Typecheck leaf and failed dependent verify, plus one selector skip. Typecheck reported runner shutdown without a TypeScript diagnostic; its Spot request reported no capacity, while the later EC2 termination reason was Client.UserInitiatedShutdown. One authorized unchanged-head targeted recovery reran only Typecheck and dependent verify; both passed. This does not claim a code fix or establish the cleanup caller. All other successful results were carried forward, not fresh remeasurements. - Current exact-head Greptile and Paperclip assessments are both 5/5 with no actionable findings; Paperclip check conclusion is neutral, but its final assessment completed and states Ship. Security checks passed and no review thread is unresolved. Full repository build/typecheck/test were not repeated locally for this three-file metadata change. A fresh actual-serving source, reporting-lock and selection guard remains required after a normal merge and before a new Submit. - No model call, eval dispatch, secret mutation, private archive download, fleet change or merge was performed by this update. The single targeted CI recovery is recorded above. Separate core PRs remain unmerged. ## Risks The discovery description may change tool choice; its effect on the missing handoff remains unproved until a new native attempt. Formatting normalization can affect outcome admission, so existing positive and negative semantics remain required. The personal secure-entry and worker phases are still unproved. The next representative set is timeline, approved-tool replay and personal secret. Company selection remains gated on personal secure-entry and worker proof. The uniform ten-case regression remains pending, and the full 244-case catalog remains unqualified. ## Model Used OpenAI `gpt-6.1-sol` through Codex. Capabilities used: source analysis, isolated mechanical integration and local code execution. The host does not expose the context-window limit. The evaluated model remains OpenCode/OpenRouter `deepseek/deepseek-v4.1-flash-1017`. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
3a2249dd57 |
fix(runner): retry proven ingress bootstrap failures (#18981)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native execution must retain its admitted model and execution authority during recovery. > - A sandbox provider can reject Runner endpoint acquisition before the Runner gets a credential. > - The old error path loses the structured HTTP status and leaves that startup failure ambiguous. > - A zero counter in an old checkpoint cannot prove that no credential was issued. > - This pull request adds a versioned producer proof and carries closed provider status through the existing RPC. > - A proved transient startup failure uses the existing retry budget. A terminal proved-unused run can settle its accounting. ## Linked Issues or Issue Description Refs #18018 and #18934. Those changes restore existing checkpoints and ready retained environments. This change covers endpoint acquisition before Runner dispatch. A search of related public PRs found no duplicate implementation. **What happened?** A provider HTTP 502 during Runner endpoint acquisition becomes an untyped worker error. Native recovery treats it as ambiguous and stops without retrying. The run's native usage receipt also stays pending after startup ends without model work. **Expected behavior** Retry a structured transient endpoint failure only when the complete current controller checkpoint proves that this exact run never issued a Runner credential. Preserve authentication, configuration, Stop, ownership and provider-progress gates. Settle zero usage only under the existing terminal accounting owner. **Steps to reproduce** 1. Use the controlled endpoint in the Daytona ingress integration test to return HTTP 502. 2. Register the actual native Runner driver and persist its bootstrap journal through the encrypted checkpoint writer. 3. Observe the old terminal ambiguous disposition instead of bounded startup recovery. **Paperclip version or commit** Reproduced on the pre-change master source. The original regression used the production native driver, a real HTTP fault, a real durable control plane and local Postgres. The new contract additionally tests the pinned Daytona SDK and real worker RPC. ## What Changed - The controller owns a v2 persisted state with retained origin and credential history. Current readers still accept v1. Older writers reject v2. Restore and checkpoint decoding do not add missing history. - The proof validates the complete bootstrap journal, all six identity fields and every delivery, credential and progress control. The server rechecks its fingerprint under the current controller claim before retry or accounting changes. - Daytona and the SDK carry a closed acquisition phase and HTTP status through the existing ingress RPC. Both public error wrappers omit response bodies and causes, including Pino cause serialization. - Only structured HTTP 429 or 5xx ingress failures can use the existing three-attempt startup budget. HTTP 401/403, configuration and unknown errors do not gain retry authority. - A terminal proved-unused run wakes existing accounting and records zero provider work. It does not create a semantic result, prove physical termination or release an unknown process/workspace hold. - The normal automatic second attempt rechecks its exact claim, frozen execution and matching complete local/encrypted journal before admitting absent remote Runner state. It keeps the original pending bootstrap command IDs. A proved unused Codex home can rematerialize sealed skills. Full raw local state is checked before streaming-event or policy projection. - Normal saved-state readers and the existing E2E journal evidence matcher accept the closed v1/v2 set. The separate legacy no-launch witness is unchanged. Technical docs describe the contracts. ## Verification Current head: `cd38f70409cdb8664e14bfbf4efa208f468a0419` (final delta adds only fixture artifact preparation). Production files are unchanged from the independently reviewed `93c8a33520` source. Full repository typecheck and build passed at `96417acb58`; the affected server typecheck also passes after the final two-line fixture delta. Focused protocol, raw-proof, real retry, SDK and RPC acceptance pass on this source. Earlier broad suite counts are listed with their limits below: - Durable Runner protocol: 187 tests. - Native executor, driver, startup/accounting and redaction: 1,211 tests, including real Postgres and encrypted checkpoint persistence. - Normal saved-state restore/projection compatibility: 21 focused cases. - SDK: 209 tests. - Standalone Daytona: 544 tests; 8 opt-in live tests skipped. - Explicit standalone `pnpm -C packages/plugins/sandbox-providers/daytona exec vitest run --config vitest.sdk-ingress.config.ts`: 6 cases with real pinned SDK 0.203.0 and worker RPC ingress acceptance. These include lookup 502/401/403/400, private-preview 502 after successful lookup/refresh, missing credentials and private-response exclusion. - E2E journal evidence: 4 cases across both supported formats, with wrong ownership, missing stimulus and unknown-schema controls. - The exact historical controller source rejects a new v2 checkpoint at decode and restore. A v1 control is accepted. This verifies the downgrade fence with the actual older parser. - Startup controls cover issued and pruned credentials, old/truncated journals, provider progress, exhausted retry accounting, and checkpoint/Stop/controller-claim races. Race cases retain the current execution and pending usage. - New exact admission controls: 19 cases through encrypted persistence and cold restore. They cover current/stale/expired claims, Stop, pause, model/trust/account changes, prior progress, issued/pruned credentials, adopted/old/truncated state, local/canonical fingerprint mismatch, hidden delivery, unsupported work and company scope. Two additional red/green cases prove that raw local stream progress and missing event policy are rejected before lossy checkpoint projection. No case mutates run, claim, result or accounting state. - Actual native executor, scheduler and Rust Runner retry: 2 cases. Codex uses its fake app server. OpenCode uses the real proxy and fake OpenCode HTTP server with the frozen DeepSeek model and allow permission configuration. Each path launches no Runner on attempt 1, starts one Runner and provider on attempt 2, reuses both original command IDs, commits one accepted result, validates the persisted local workspace, records nonzero synthetic native usage and settles accounting once. No zero-work accounting flag survives. The controlled SDK and protocol tests use local source, isolated PostgreSQL, a private HTTPS preview and fake provider servers on macOS arm64 with Node 24.22.2. They create no hosted sandbox and make no model call. The root workspace intentionally omits standalone provider dependencies; the explicit SDK target fails if its pinned dependency is absent and has no conditional skip. They do not qualify hosted rollout or repair an old unmarked checkpoint. The first full local typecheck found missing checkout SDK links from an install with scripts disabled; the repository link helper repaired that setup. Current-head Linux CI typecheck and build pass. Independent final review checked 12 source scopes and six raw-proof controls with no remaining finding. Fresh Greptile review is 5/5 with no unresolved thread. The full Runner CI suite found two older control-peer test doubles missing the new proof method. Both now return no proof, with all 27 permission, ordering and cancellation assertions retained and passing. The new Cargo-dependent native retry suite uses the existing Rust-cached CI lane. All 39 lane coverage controls pass and prove complete, duplicate-free suite coverage. The first full test run stopped at an admitted chat Retry polling failure. CI also found a stale retained-allocation fixture, now repaired on master by #18984, and a canceled runner job. A real SDK suite initially relied on a dependency absent from the root workspace; the explicit standalone target now owns that acceptance. The separate Product E2E support run had 12 failures in four untouched files. All 12 reproduce on the original clean base with independent dependencies, compiled base source and verified workspace resolution. These concern workflow expectations, cleanup timeout arguments, a profile snapshot and operating receipts. They are not passing acceptance. The later full local repository test command stopped at a 15-second timeout in the unchanged GitHub pre-allocation retry `controller` case. That exact case passed in isolation with unchanged assertions. The broad Runner run initially omitted `test:typescript:prep` and lacked debug fake-provider binaries. Required preparation then passed 370 + 44 package contract cases (11 skipped); the prepared full package run reported 4,932 passed, 9 failed and 13 skipped. All failed families then passed with unchanged assertions in isolation: 8 ACPX controller tests and 2 credential controls. The paired original-base ACPX control also passed. The full-run failures remain recorded; they are not passing full-suite evidence. The first Rust-cached native CI run found a missing generated OpenCode proxy in a fresh checkout. The fixture now runs the existing verified provider-entrypoint builder. Removing the generated proxy reproduces the exact ENOENT before the fix; both real retry cases pass after the fix from the same absent-artifact state. All current-head CI gates pass, including the aggregate verification check. Linux native CI passes all 10 suites: 177 tests pass and 14 opt-in cases skip. Both real provider retry cases pass. The full Runner package passes 297 suites and 4,950 tests, with 1 suite and 10 tests skipped. The CLI workspace groups pass 4,762 and 286 tests. The first CI attempt had Runner shutdowns in the Runner package, Canary and three general server jobs. One unchanged CLI shard also hit child-server startup and dense ZIP deadlines. Both CLI files passed unchanged in isolation: 18 tests. One bounded rerun of unsuccessful jobs and their dependents passed. The original failures remain recorded. ## Risks - Older controllers reject newly written v2 state on downgrade. Current controllers retain v1 restore support, but old or unmarked state cannot gain this new proof. - An older plugin that omits structured ingress evidence remains held. The server and provider must use the new producer contract for this recovery path. The Docker `cloud` target builds the SDK, server/Runner and standalone Daytona producer from one source tree. The default `production` target does not build the standalone producer. Hosted rollout must confirm the new image and active image-owned plugin worker; a registry version alone does not prove its source. An npm-installed producer needs an explicit matching artifact upgrade. - This prevents future proved startup failures. It does not release historical accounting based on an old checkpoint, change retry budgets, replace a model or weaken sandbox isolation. - Zero model usage is separate from process termination, saved files and resource release. Those owners keep their existing gates. ## Model Used OpenAI Codex with tool use and code execution. This runtime does not expose the exact backend model ID or context window to the agent. Read-only independent agents reviewed checkpoint compatibility and SDK/RPC error boundaries. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (affected suites; full local run limits are recorded above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>canary/v2026.1104.0-canary.11 |
||
|
Devin FoleyandClaude Opus 5.5
|
6a45ea479e |
fix(heartbeat): retry failed legacy lease renewals until the deadline (#17851)
<!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/52/changes/paperclip/52 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A legacy agent run keeps a durable controller lease. The server renews the lease every 10 s. The lease lasts 60 s. > - The lease watchdog stops the run when it loses the lease, because another controller can then take the run. > - #17032 moved renewal to a private pool. It also retries a renewal that hits its statement timeout. But a renewal that fails with any other error still stops the run at once. > - Staging still shows this failure after #17032. A legacy run made 186 good renewals over 31 minutes. Then one renewal failed with an error 23.9 s after the last good renewal, and the watchdog stopped the run immediately (`controllerLeaseLossCause=renewal_error`). The lease still had about 36 s left. > - This pull request retries a failed renewal until the lease deadline, the same as a timed-out renewal. > - The benefit is that one dropped database connection no longer cancels a healthy run. The 60 s lease deadline still bounds the retries, so the ownership guarantee does not change. ## Linked Issues or Issue Description Refs #17032 Refs #16084 **What happened?** A legacy agent run on staging was cancelled mid-turn. The run was healthy. The cancellation evidence was `cancellationSource=controller_lease` and `controllerLeaseLossCause=renewal_error`, with `controllerLeaseRenewalCount=186` and `controllerLeaseSinceRenewalMs=23913`. The release already contained #17032. **Expected behavior** One failed lease renewal must not stop a run while the lease is still valid. The watchdog must try the renewal again. It must stop the run only when the lease deadline passes or when the database rejects the renewal. **Steps to reproduce** 1. Start a legacy run so that `watchLegacyControllerLease` watches it. 2. Make one renewal query fail with a connection error, for example terminate its Postgres backend with `pg_terminate_backend`. 3. See that the watchdog aborts the run immediately with `renewal_error`, although the lease has not expired. ## What Changed - `watchLegacyControllerLease` retries a renewal that fails with an error. It waits 1 s between attempts (`LEGACY_CONTROLLER_RENEWAL_RETRY_DELAY_MS`). The existing lease deadline stops the retries. - A retried renewal is safe. The UPDATE is idempotent, and it still requires this controller's boot ID and an unexpired lease. A rejected renewal (no row updated) still stops the run at once with `renewal_rejected`. - If the deadline passes while the attempts still fail with an error, the cause stays `renewal_error`. A statement timeout counts as a stall, so it stays `renewal_deadline`, as before. A good renewal clears the error state. - `deadlineLateMs` is now recorded whenever the deadline ends the lease, also for `renewal_error`. - Tests: - New embedded-Postgres test: it terminates the renewal's own backend while the renewal waits on the run row. The renewal retries on a new connection and keeps the lease. Without the fix, `assertOwned` rejects. - New unit tests: one failed attempt is retried and the lease is kept; a stall after a recovered error is still `renewal_deadline`. - The existing `renewal_error` test now checks that a renewal that always fails is retried and is reported as `renewal_error` at the deadline, with its timings. - `doc/run-log-events.md` describes the retry and the updated `renewal_error` and `deadlineLateMs` meaning. ## Verification - `server/src/services/legacy-controller-lease.test.ts` with embedded Postgres: 23 tests pass. - With the original `legacy-controller-lease.ts`, 4 of these tests fail. One of them is the real-Postgres test that terminates the renewal connection. - `server/src/services/heartbeat/run-completion.test.ts` and the `run-cancellation` tests: 35 tests pass. - A typecheck of the changed files reports 0 errors. CI runs the full typecheck. ## Risks - Low risk. The change is limited to the legacy lease watchdog. - A run with a permanently failing renewal now stops at the lease deadline (at most 60 s after the last good renewal), not at the first error. Another controller cannot take the run before the lease expires in the database, so ownership stays safe. - A non-transient error (for example a code bug in the query) is retried once per second until the deadline. The cause label is still `renewal_error`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, code search, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
DottaandPaperclip
|
8f06bd4e17 |
fix(e2e): retain workspace packages in rolling app copies (#18982)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A rolling deployment must preserve running native agents and their work. > - The source test lane starts separate copies of the compiled app. > - The development Runner entry imports source from the workspace packages directory. > - That directory was absent from each copied app, so its supervisor could not start. > - This pull request preserves the package link beside the existing CLI link. > - The test lane can then exercise real controller replacement. ## Linked Issues or Issue Description **What happened?** The source rolling-deployment supervisor failed with a missing Runner module before starting a paid agent. The copied app had its compiled server, dependencies and CLI link, but lacked the workspace package path used by the development Runner entry. **Expected behavior** The copied app resolves its development dependencies and reaches controller replacement. **Steps to reproduce** 1. Build the server and UI from source with the development Runner entry. 2. Run `tests/runner-e2e/controller-replacement-smoke.ts` with disposable PostgreSQL and S3. 3. The old copy fails to resolve the Runner package. The fixed copy completes the supervisor smoke. No duplicate PR or issue was found in the exact-path search. This is a test-harness fix within the existing rolling-deployment lane. ## What Changed - Link the checkout's `packages/` directory into each copied source app. - Document the shared dependency links and the limits of source-lane evidence. ## Verification - The missing-module failure was reproduced before the link was added. - The same supervisor smoke passed after the fix and confirmed cleanup. - The ten-check non-agent rolling smoke passed with disposable PostgreSQL and S3. - A fresh strict Codex rolling browser case passed. It kept the same running process, native session and sandbox lease, acquired a new controller generation, finalized the workspace, and continued from the saved project. - An earlier paid attempt failed the strict tool oracle because the agent's own soak script omitted the argument delimiter for hyphen-prefixed input. That failure remains recorded; the verifier and task contract were unchanged for the passing attempt. - Server TypeScript compilation passed in the source test checkout. Full repository typecheck, tests and build are required in CI for this clean PR head; they have not been repeated locally in this dependency-free checkout. - Exact head `d8addaa7723af5a013489e5bb6331dd1021dc6be`: all CI checks passed or were intentionally skipped; Greptile 5/5 with no review threads. Earlier cancelled jobs and the old recovery-fixture failure remain retained; rebasing onto its existing master fix passed cleanly. ## Risks Low production risk: only source test setup and its documentation change. Each compiled app still has a separate cache and home. Its workspace dependencies are shared, so this does not qualify a released container or Railway deployment. ## Model Used OpenAI Codex, exact model ID `gpt-6.1-sol`, with reasoning, tool use and code execution. The host does not expose the context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal ticket id or instance-derived details - [x] I have run tests locally and they pass, with the earlier material failure retained above - [x] I have added or updated tests where applicable: the existing supervisor and live rolling browser lane exercise this path - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
Devin FoleyandPaperclip
|
edb9d29942 |
fix(recovery): preserve Retry authority and reopen cross-task cases (#18997)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Tasks keep execution identity and saved workspace files separate from control-plane updates. > - A scanner can have permission to reopen a case while its run cannot restore that case's files. > - Retry can also follow a preparation failure that has no recovery-action record. > - Both paths can accept valid work but leave the target unable to run. > - This change restores files under the target executor and saves admitted Retry authority on the canonical receipt. > - Existing workspace, user, source and successor checks still apply. ## Linked Issues or Issue Description **What happened?** An authorized agent updates another closed task. The route tries to restore the target workspace under the caller's run identity and returns `403 Task workspace run identity is unavailable`. A separate failure occurs when a normal board Retry follows a pre-provider preparation failure. Admission has no recovery-action row on which to save continuation authority. Dispatch then fails with `continuation_user_authorization_missing`. **Expected behavior** Accept the authorized task update. Restore its terminal workspace when the target's admitted executor starts. An admitted Retry must retain its exact dispatch authority even when the predecessor has no recovery-action record. **Steps to reproduce** 1. Complete a task with an isolated workspace. From a different running task, use an authorized comment or PATCH with `resume: true` on that closed task. 2. Separately, fail native preparation before provider work starts, with no recovery-action row. Submit board Retry, admit its successor, and build the execution continuation. 3. Before this change, the first path returns 403 and the second rejects its admitted authority. **Paperclip version or commit** Reproduced against `e968131efa7e63d321f3bcca8d47a62513c2efb4`. Rebased onto `afbc5a885b5`. **Deployment mode** Server-managed native task execution. Tests use real PostgreSQL and temporary workspace files. Related PRs: #16927 covers terminal workspace races. #17916 retains exact native export sources. Neither covers these route and Retry receipt failures. A GitHub search found no duplicate. ROADMAP.md was checked; this is a recovery bug fix. ## What Changed - Defer workspace restoration only for authorized cross-task comments and updates from a live run. Checkout keeps its existing workspace identity checks. - Restore the target's selected terminal workspace under its own admitted run. Preserve its saved files and existing source checks. Leave manually archived workspaces unchanged. - Store server-created continuation authority on the canonical admitted Retry receipt when no source recovery-action record exists. Existing action records remain authoritative. Read the fallback only after matching the company, task, agent, user, source, generation and successor. - Add route and PostgreSQL regression tests. Document both paths. ## Verification - Red: both cross-task route tests returned the observed 403 before the route fix. - Red: the target executor could not reuse its archived terminal workspace before the restoration fix. - Red: an admitted preparation Retry with no recovery-action row failed with `continuation_user_authorization_missing` before the receipt fix. - Green: 419 recovery and continuation tests pass, plus 183 focused route, environment and Retry tests. The assembled heartbeat test starts the agent in the same terminal worktree and verifies both its unpushed commit and uncommitted file bytes. It fails before the restoration fix. - Negative cases reject wrong Retry runs, users, source runs, generations, unadmitted receipts and missing authority. The original failed run stays unchanged. Workspace tests verify saved file bytes. - Full `pnpm build`, `pnpm -r typecheck`, `pnpm check:module-boundaries` and `git diff --check` pass. - The full suite found an existing assembled chat authority regression: clearing a recovery-action record could be bypassed by the new fallback. The fix now writes the fallback only when that source has no action record. That assembled regression passes with the original test unchanged. - Full `pnpm test:run` is running again. Its first attempt also hit an unrelated ancestor skill-root lookup for a missing Slack skill. The rerun uses a temporary untracked link to the repository's own bundled skills. This link is excluded from the PR. Full-suite completion and hosted CI are pending. - Hosted CI has multiple runner interruptions. E2e shard 1/8, server shards 3/12, 8/12 and 12/12, and workspaces-a shard 1/2 show runner shutdown signals. E2e shard 5/8 and server shards 1/12 and 2/12 have GitHub annotations reporting lost runner communication; their logs return 404. The `ci / verify` assertion failed because the general-test matrix was cancelled. GitHub rejected job and failed-workflow reruns while this workflow is still running. Rerun the interrupted checks after it settles. - Greptile reviewed exact head `a1e3929aedb1ee3bfcd001f4ebb1510106bf4070` at 2026-11-04 17:37 UTC: 5/5, with no new blocking findings and no unresolved review threads. ## Risks - A deferred workspace restore can still fail at target admission. The run keeps the existing workspace failure path; this change does not bypass file recovery or provider stop checks. - Retry receipts now store server-created authority. Existing action receipts remain valid. No schema migration is required. - Local regression tests establish the software behavior. The new revision has not been deployed to a hosted instance. ## Model Used OpenAI `gpt-6.1-sol` through the Codex harness in T3 Code, with xhigh reasoning, repository inspection, tool use and code execution. The context-window size is not exposed by the runtime. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
039e0dad2c |
Connect TestOps protocol plans to verified GitHub attempts (#18985)
## Thinking Path > - Paperclip lets people coordinate agent work through a governed control plane. > - Runner protocol evals test real provider sessions against a seeded mock control plane. > - The saved TestOps protocol plan has no GitHub executor. Its dashboard shows historical results. > - A current one-case attempt also failed before any tool call or provider output. Its retained public-safe snapshot withheld the error cause. > - This pull request connects selected cases to the existing trusted workflow and retains closed operator failure facts. > - Each selected case receives a current result. Missing evidence remains broken. Existing assertions and machine grades remain unchanged. ## Linked Issues or Issue Description **Current behavior** The saved protocol plan cannot start a GitHub qualification attempt. Historical dashboard results can be mistaken for current execution. A failed provider turn can also lose its failure category when the live snapshot applies public evidence redaction. **Proposed behavior** The existing workflow accepts TestOps service inputs, validates exact case identities before provider work, runs the selected cases, and verifies their published statuses. Eval artifacts retain only validated failure phase, category, HTTP status, and retryability. **Reason and benefit** Operators can run the standing plan from TestOps and inspect current, source-bound results. A targeted pass cannot stand in for full-plan coverage. A failed startup can be diagnosed without publishing provider messages or request identifiers. ## What Changed - Add optional TestOps selection and protected reporting jobs to the existing Runner protocol workflow. - Keep the current actor, source, provider credential, and controlled accounting gates. TestOps retains the explicitly selected controlled or legacy mode and a separate 41-case Claude roster. Controlled mode preserves its existing reviewed policy, immutable bindings, tracing and zero-retry admission. Its paid workers explicitly use GitHub-hosted runners, matching the current manifest admission contract even when the legacy AWS fleet is enabled. - Bind the exact standing plan, workflow, job run, source revisions, selected cases, and saved statuses. Accept only verified workflow URL aliases. - Create a closed Allure projection inside the report job. Transfer only fixed case identities, check verdicts, numeric usage and statuses to the reporting environment. Controlled raw/chat/controller evidence remains private. - Publish one result for every selected case. Retain missing execution/report evidence as broken. Reject unsupported passes, model mismatch and source drift. Preserve the evidence schema literal, stop publication on cancellation, and retain closed API error codes/statuses. - Tag full selections separately from targeted attempts. Add bounded ingestion observation. Apply the exact TestOps filter before the unchanged controlled provider/cell-capacity check, so a one-cell registration fits a one-cell reviewed policy. - Retain up to eight eval-only closed provider failure records before board evidence redaction. Forward the original notifications unchanged. - Add selection, reporting, privacy, and admission regression controls. Document scope and remaining acceptance. ## Verification - Direct Node protocol publication, projection, metrics, and security tests: 56 passed. Focused controls exercise both mode admissions, missing controlled policy/retries/source rejection, closed result transfer, exact metadata, and missing-result handling. - `actionlint .github/workflows/runner-protocol-live-evals.yml` and `git diff --check` passed. - A proposed one-cell controlled policy passed the actual hosted validator with a synthetic prepared request and zero provider calls. The actual 41-case catalog filter selected case194/get-task-context and the real hosted admission accepted it under maxCells1. Negative controls rejected the unfiltered41, unsupported provider, source, selection, origin, expiry and lock drift. This does not verify provider auth, a runtime build or TestOps publication; final merged source/lock binding is still required. - The actual diagnostic TypeScript module passed positive, negative, secret-exclusion, and unchanged-notification controls through the existing tsx runtime. A local actual-CLI startup serializer probe passed with closed facts, secret exclusion, exit status and read-only fixture cleanup. This used an isolated resolver into the existing dependency cache; canonical exact-dependency CI remains required. - Prior-head CI passed typecheck, Runner static/Rust/native-server checks and 4946 Runner package tests. One new serializer fixture failed during Linux read-only directory cleanup; this head applies the existing neighboring cleanup pattern. Unrelated chat E2E and CLI import timeouts remain retained. Latest-head required gates are pending. - Original current-source one-case attempt: https://github.com/paperclipai/paperclip/actions/runs/41860234840 . It failed with no tool calls or output. That failure remains retained and is not a passing qualification claim. - Diagnostic candidate attempt: https://github.com/paperclipai/paperclip/actions/runs/41863410602 . It retained the new closed field but still failed before output with category unknown. This disproves the claim that typed retention alone identifies the cause. - The earlier passing run https://github.com/paperclipai/paperclip/actions/runs/40627274706 used controlled Braintrust routing. Both current failures used direct legacy credentials, a different auth path. Next acceptance is one explicitly controlled case with a reviewed current policy, followed by TestOps registration and a fresh full plan. None is claimed complete by this PR body. ## Risks The TestOps contract currently binds the existing 41 cases and one recorded Claude Sonnet 5 profile. Catalog changes fail closed until reviewed. This does not qualify other providers. Admission and execution qualify only the selected accounting mode; a controlled pass does not prove direct credential health. Model-request readiness retains pending operating reconciliation. TestOps ingestion can fail independently; persisted results are checked, and missing evidence is never accepted as a pass. The new failure field contains closed facts only; raw messages, identities, credentials, and tool payloads are excluded. ## Model Used OpenAI Codex, GPT-6. The exact hosted model ID and context window are not exposed in this session. Used reasoning, repository inspection, code editing, and tool execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used and the available capability details - [x] I have checked ROADMAP.md and confirmed this does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs - [x] I have described the issue with the enhancement template field labels - [x] I have not referenced internal or instance-local Paperclip issues - [x] My branch name describes the change and contains no internal ticket ID - [x] I have run the available targeted local checks and reported the remaining checks - [x] I have added relevant positive and negative regression controls - [x] I have updated the authoritative protocol documentation - [x] I have considered and documented the risks - [ ] All required Paperclip CI gates are green on the latest head - [ ] Greptile is 5/5 with no open actionable feedback on the latest head - [x] I have addressed all current review feedback --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
b99dd57c82 |
test(evals): run Product regression from TestOps (#18983)
## Thinking Path > - Paperclip manages agent work through governed product workflows. > - Product E2E tests check those workflows through the real app and runner. > - The saved Product regression plan has 20 imported case identities. > - The plan has no live executor, so historical baseline rows do not qualify its current behavior. > - The existing paid full-stack workflow already owns all 20 required journeys. > - This change binds TestOps selection and reporting to that workflow without changing the product oracle or paid admission. > - Maintainers can run the plan and inspect current case verdicts and retained evidence. ## Linked Issues or Issue Description **What happened?** The standing Product regression plan has no executor. Its dashboard points at a historical 355-result snapshot across several suites and revisions. **Expected behavior** A TestOps plan run must select the intended existing full-stack cells, execute them on GitHub, and report their original case IDs, assertions and source revision. Missing evidence must remain a visible failure. **Steps to reproduce** Open the Product regression plan and inspect Executors. There is no live job. The existing full-stack workflow accepts exact cell IDs but has no TestOps selection or reporting path. **Paperclip version or commit** The branch includes reviewed master correction |
||
|
DottaandPaperclip
|
6364fa7327 |
fix(heartbeat): wait for retry predecessor cleanup (#18989)
## Thinking Path > - Paperclip manages agent work and preserves its task ownership. > - Heartbeat admission controls when a queued run can start. > - A failed run can still own workspace saving and environment cleanup. > - Automatic recovery can transfer the task execution lock to its queued retry before that cleanup finishes. > - The old queue check then sees the retry itself and misses the predecessor's remaining work. > - This pull request checks the exact retry predecessor before claiming the queued run. > - Existing cleanup notifications can then resume that same run through normal admission. ## Linked Issues or Issue Description Refs #18898 and #18691. The first repair covers canonical connected-task retries with a native source. The second covers live remote workspace writers. This change covers ordinary automatic continuation when the predecessor failed before native runtime selection. The search found no open repair for that path. **What happened?** A preparation failure saved a terminal legacy run row while its remote workspace save and environment cleanup were still active. Generic continuation created one queued successor and moved the task execution lock to it. The queue checked that current lock but did not check the successor's predecessor. The new run started and failed at the existing unfinished-writer guard. **Expected behavior** Keep that same successor queued until its exact predecessor finishes execution and cleanup. Check company, agent and task identity before using retry provenance as a wait condition. Preserve Stop, reassignment, failed cleanup and independent recovery gates. **Steps to reproduce** 1. Save a failed legacy preparation run with an active environment lease. 2. Queue its ordinary automatic continuation and bind the task execution lock to the successor. 3. Claim the queued run before the predecessor finishes cleanup. 4. The old code starts the successor. The fixed code retains it in the queue. 5. Commit successful cleanup through the existing cleanup owner. The ordinary dispatcher starts that same run once. **Paperclip version or commit** The causal regression fails on master `86814c6550f52e99ede150a883bdc773cbb540e3`. **Deployment mode** Source checkout on macOS arm64 with disposable PostgreSQL, the assembled heartbeat service and the production work coordinator. The adapter boundary is controlled. No hosted provider replay or deployment is claimed. ## What Changed - Check the current task execution owner and the retry's exact predecessor under the existing task claim lock. - Use the saved native task binding, ordinary task binding and historical context fallback in that order. Ignore unrelated retry provenance. - Reuse the existing live-executor and pending-cleanup checks. Reuse committed cleanup notifications and the existing dispatcher. - Add real PostgreSQL claim controls and assembled cleanup-event acceptance. Keep the original source failure, saved files and current task ownership intact. - Document the queued predecessor-cleanup rule. ## Verification - RED: the unchanged queue claimed the successor and wrote a new controller lease while its legacy predecessor retained an active remote environment lease. The regression expected the successor to remain queued. - GREEN: the same regression passes after the change. The complete queue suite passes all 71 tests. - The assembled heartbeat suite passes all six tests. The added cases cover one continuation after committed cleanup, saved-file reuse, final task lock release, board Stop, reassignment and failed cleanup. - Scope controls cover a deleted source after scan, another company, another task, another agent, canonical native task binding and the existing current-owner hold. A failed cleanup and a still-live local executor retain the wait. - Node 24.19.0 and pnpm 9.15.4. Module boundaries and `git diff --check` pass. - Initial fixture attempts failed on database constraints and observer timing. The final fixtures use valid immutable bindings, the board Stop attribution and a completed real dispatcher call after reading the exact committed lease state. Production ownership guards were retained throughout. - Full low-memory repository typecheck passed at `5cddb49322`. The sync to master `04a24f4e06` adds only the independent retained-allocation test fixture from #18984; production code is unchanged. - Full `pnpm build` passes on current head `c7c2bcf08ffd05be5ce714683ae1dcfe6f5b49f1`. - The post-sync run executes all 80 behavior assertions successfully across the queue, assembled cleanup and updated upstream retention suites. Its queue `afterAll` then exceeds the existing 30-second database shutdown hook. That failed suite result is retained. The controlled post-build queue rerun passes all 71 tests with normal teardown. The earlier shutdown cause remains undetermined. Its owned queue-test PostgreSQL process has exited. The separate local default `pnpm test:run` was intentionally stopped after full unfiltered CI passed on the exact head. Four local chat partitions completed before that stop; this is not a local full-suite pass. The verified owned process group and its PostgreSQL fixture have exited. Logs and retained fixture data remain available. The timeout has not been increased. - Full unfiltered CI passes on current head `c7c2bcf08ffd05be5ce714683ae1dcfe6f5b49f1`: all 12 server shards, all six internal chat partitions, all nine serialized server shards, the workspace groups, native Runner lanes, typecheck, build and browser E2E. Both required [ci / verify](https://github.com/paperclipai/paperclip/actions/runs/41866478617/job/126711606613) and [ci / e2e](https://github.com/paperclipai/paperclip/actions/runs/41866478617/job/126711263146) pass. The file selector is skipped, so this is the full CI test matrix. The head has 55 successful checks and three expected skipped checks. The ready-for-review contributor trust and security scan also pass. - Current-head Greptile is 5/5, its check is successful, and no inline review threads remain. The ordinary review check also passes. No merge, deployment or original task retry was performed. ## Risks - A predecessor with unfinished cleanup keeps its successor queued. Its existing cleanup owner remains responsible for settlement. - A task claim reads at most one additional predecessor row. A matching source also uses the existing ownership and cleanup queries. The change adds no timer, schema, credential, provider or model policy. - A committed cleanup signal only requests normal admission. Current ownership and independent execution gates still apply. ## Model Used OpenAI GPT-6 through Codex, with reasoning, code editing, tool use and PostgreSQL verification. The session does not expose a more specific serving model ID or context-window limit. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass — focused suites pass; the separate local full-suite attempt is explicitly incomplete above - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
aaea05e158 |
fix(native): preserve backups when completed warm sessions retire (#18986)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native Runner keeps provider history when a warm session ends. > - Object storage must preserve that history before the sandbox is released. > - A completed turn releases its live controller lease. > - The backup owner lookup rejects that released lease and stops the next task preparation. > - This pull request uses the existing completed-run checkpoint fence for backup storage. > - The next turn can prepare after the previous provider state is saved. ## Linked Issues or Issue Description Refs: #18932, #18011, #18984. **What happened?** A warm Codex turn completed and kept its provider session alive. The next turn stopped during preparation with `native_harness_storage_owner_missing`. The backup lookup required a live lease after normal completion had released it. **Expected behavior** Save the completed session under its retained controller generation and committed result. Keep the existing owner and successor checks. Do not start another provider turn during the save. **Steps to reproduce** 1. Run a warm native session with object-backed provider backups. 2. Commit the first turn and release its controller lease. 3. Start the next task preparation and retire the previous session. 4. Observe the backup owner error before provider work starts. **Paperclip version or commit** The regression is present at `86814c6550f52e99ede150a883bdc773cbb540e3`. **Deployment mode** A hosted server with a remote native Runner and S3 storage. ## What Changed - Accept a released backup owner only when its exact controller record has a committed result. - Reuse the checkpoint owner policy for same-controller preparation and native successors. - Recheck the controller generation, committed result, and released state after upload. - Preserve live-owner rejection and the rule that a late upload cannot replace a newer conversation backup. - Add real PostgreSQL storage and restore regressions. Document the completed-run backup rule. ## Verification - On 2026-11-04, two new regressions failed on the unchanged source with `native_harness_storage_owner_missing`. These covered an idle completed session and a same-controller preparing successor. - After the fix, 54 selected tests passed: 30 durable backup tests and 24 existing early-retirement checkpoint tests. After the fixture-only upstream sync, all 58 selected storage, checkpoint, process, and affected fixture cases passed again. - The storage cases use real PostgreSQL, encrypted archives, object transfer, and restoration. They cover invalid completion evidence, foreign or expired successors, active predecessors, changed generations, changes during upload, and newer backup publication. - Command: `pnpm exec vitest run server/src/services/native-runtime/native-harness-backup-storage.integration.test.ts server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts -t 'released|backup|early retirement|durably retires an accepted warm Runner'`. - An independent source review found no outstanding code issue. - Full repository typecheck passed with `node scripts/typecheck-low-memory.mjs` before the upstream fixture-only sync. Production source is unchanged across that sync. - An assembled process probe passed with the Runner binary built from this head and a fake Codex protocol peer. It started real processes, committed a result, released the source lease, claimed the next controller, and completed the source stop, drain, and suspend commands. It recorded one provider turn start and kept the task bound to its successor. - Command: `pnpm exec vitest run server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts -t 'durably retires an accepted warm Runner'`. - CI exposed the old allocation-readiness test setup in three cases. The branch now includes the merged #18984 fixture correction. The same three cases passed locally after that sync. - An earlier CI test shard was cancelled after its runner received a shutdown signal. Its log was retained before one rerun. Another old-head shard reported failure with no retrievable job log or annotations; its cause remains unknown. Current-head CI must pass independently. - Local `pnpm build` passed at this head. All 1,904 chat cases passed across the supported local shards; the earlier shards used byte-identical source before the fixture-only sync. The interrupted default command is retained and is not reported as a pass. - The local bulk server run had separate enrollment acknowledgment, environment-run status, monitor-settle, idle-history teardown, and repository-rehome failures. In isolation, idle-history passed all 75 tests and the unchanged repository cases passed with their original limits (4.9s large files, 6.4s refs). The remaining four local failures reproduced on unchanged master `04a24f4e06` under the same environment and limits. Their causes remain unproven. Logs and the baseline comparison were retained; the local full suite is not claimed as passed. - Current-head CI first hit the unchanged repository-rehome large-file case's 15s timeout. The same isolated case passed in 4.9s. After source/call-path comparison, one bounded failed-shard retry passed all 115 executed suites; no assertion or timeout was changed. The original timeout remains recorded. - Local verification uses macOS 26.5.2 arm64, Node 26.4.0, pnpm 9.15.4, PostgreSQL 18.1, and Cargo 1.97.1. The current tested head is `622f69f76ff68c71eacdcbcbc6ab4d4c6eb83bf9`. - The full unfiltered [PR workflow](https://github.com/paperclipai/paperclip/actions/runs/41868587974) passed at `622f69f76ff68c71eacdcbcbc6ab4d4c6eb83bf9`: all 12 general server, 3 chat, 9 serialized, workspace, native Runner, build/typecheck, E2E, and final verify lanes. The file selector was skipped, so this was the complete fallback matrix. - Greptile reports 5/5 on this exact head with no open review threads. Disposable local test processes were cleaned up and the source tree is clean. - The real hosted Codex and S3 workflow has not been rerun because the local verification environment has no remote provider credentials. The original task remains unchanged. ## Risks The main risk is a stale controller publishing provider history after a new owner starts. Both storage transactions keep the company, run, boot, and generation fence. A released owner must keep its committed result. Existing successor checks and the newer-backup publication guard remain in force. The live checkpoint writer still requires a non-null lease. No migration or API change is required. ## Model Used OpenAI Codex, GPT-6, with reasoning, repository tools, and code execution. The exact serving model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass (focused checks; separate baseline broad-suite failures are disclosed above) - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
afbc5a885b |
fix(workspaces): save partial and shallow Git checkouts offline (#18990)
## Thinking Path > - Paperclip manages AI agents and their workspaces. > - A completed remote run must save its files before finalization can release its source. > - A shallow Git mirror uses upload-pack and can omit objects that exist only in the index. > - A filtered source can also fail when that mirror requests blobs the source never downloaded. > - Fetching missing history is not required to preserve an existing partial checkout. > - This change saves the available object stores and their existing shallow, sparse, and promisor state. > - The benefit is an offline save that preserves accepted work and remains readable by the existing archive reader. ## Linked Issues or Issue Description Refs #16727, #18705, and #18935. These changes cover managed clones, streaming capture, and safe diagnostics. This change repairs Git snapshot contents. **What happened?** Remote workspace capture invokes a mirror clone before it saves the archive. A shallow filtered sparse checkout with a missing promised blob fails that clone and surfaces `remote_workspace_operation_failed`. An ordinary shallow checkout can also lose an index-only object during the mirror transfer. **Expected behavior** Capture preserves the objects already available in the checkout, including staged work. It preserves partial, shallow, and sparse semantics without remote hydration. Corrupt or unsupported state must keep its source and receive no successful save receipt. **Steps to reproduce** 1. Create a real Git repository with two paths and commit history. 2. Clone it with `--filter=blob:none --depth=1 --sparse`, then select one path. 3. Stage a change, edit it again, and add an untracked file. 4. Remove access to the original repository and capture the workspace. 5. The base capture fails before transfer. The changed capture restores the available work after the original source is removed. **Paperclip version or commit** Reproduced on `86814c655`. The same transport/archive sources are present in `76e43852a`. ## What Changed - Preserve existing object files for shallow and partial repositories, including their original promisor pack markers. - Preserve shallow boundaries and the selected sparse settings and patterns. - Validate available object contents with full Git fsck. Reject unknown corruption instead of creating new promises. - Deny Git network protocols during capture. Keep the ordinary mirror's explicit local file transport. - Exclude ambient Git location, index, object-store, config, and template overrides. Disable templates for init and clone. - Keep the existing archive format, source fences, path guards, credential allowlist, and failure retention. - Add real command and reader regressions for full, shallow, filtered, sparse, corrupt, and unsupported repositories. ## Verification - Red: the compiled Node 24 transport completed helper installation, then failed capture with the same generic operation error and no transfer. Dirty source bytes remained intact. - Red: the new full/shallow/partial transport-to-reader cases exposed both filtered upload-pack failure and a missing staged-only object. Independent review then reproduced accepted corrupt object bytes and ambient template config/hook copying. - Green: all 110 transport/archive cases pass after those repairs. They cover staged, dirty, and untracked work; ordinary and forged-promisor corruption; malformed filters; alternate stores; unsafe links; credential redaction; and template isolation. An initial linked-worktree config regression was fixed before this complete passing run. - A compiled new capture restored through both the unchanged earlier reader and the new reader after source removal. HEAD, status, work bytes, sparse/shallow state, and the same missing promised object were preserved. Full Git fsck passed. No remote protocol was permitted. - Full repository `pnpm -r typecheck` and `pnpm build` pass at `91ca00a8327216def91303e19940e92ffe07c08f`. The build stamp names that exact head. - The [complete current-head CI run](https://github.com/paperclipai/paperclip/actions/runs/41865961403/attempts/2) is green: all 58 checks are terminal with no failure or pending check. Three expected skips and one neutral review remain those states. - Preserve the [first CI attempt](https://github.com/paperclipai/paperclip/actions/runs/41865961403/attempts/1): one browser case reached its 5-second heading deadline while the conversation pane was loading; a chat worker reported shutdown; another browser worker lost communication and its log blob was absent. A failed-job-only retry passed at the same source head. No test deadline, assertion, or source change was made for that retry. - The complete local `pnpm test:run` passed all 1,904 chat cases across six partitions. Its later general-server phase timed out two unchanged tool-access cases at 15 seconds. That known-red command was stopped before completion. Both cases then passed unchanged in a fresh isolated process. The original timeout cause remains unknown; this is not a full local-suite pass. - [Greptile reports 5/5](https://github.com/paperclipai/paperclip/pull/18990#issuecomment-6635931211) at the current head with no actionable finding. The product review's loose-object cost advisory was [answered with local measurements and resolved](https://github.com/paperclipai/paperclip/pull/18990#discussion_r4470261833). There are no unresolved review threads. - Original service acceptance remains unverified. Recorded checkout text is mechanism-compatible narrative evidence, not proof of its physical Git state or original failure cause. This PR includes no deployment or additional live recovery. ## Risks - Full object validation and copying available metadata use CPU and temporary space proportional to the local object store. - The offline path rejects alternate object stores and unsupported partial filters. Such sources remain protected. - Promised objects remain missing. Later work still needs ordinary repository access to request them. - There is no archive format or schema change. Existing readers restore standard Git files, and existing full snapshots remain covered. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, repository tools, shell execution, and local verification. The host does not expose a more specific serving model ID or context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
e968131efa |
test(evals): pin reviewed outcome and native handoff repairs (#18976)
## Thinking Path > - The catalog must measure the original human requests through ordinary product controls. > - The latest four-case run exposed two oracle false negatives and a real suppressed native handoff. > - Preserve those original failures and fix only their demonstrated causes. > - Keep the reviewed company-selector successor and add the accepted timeline, approved-replay and personal-completion corrections. > - Select native standing-chat terminal answers by authority while preserving governed waits and ordinary-task behavior. > - Bind the resulting immutable candidate and its three changed oracle versions through the existing manifest and descriptor snapshot. > - The next personal-secret attempt must prove the complete secure-entry and worker journey before company selection or a final ten-case regression. ## Linked Issues or Issue Description Refs #18971, #18967, #18968, #18972, #18977. Follows #18974. The latest four-case run retained original failures for timeline, approved tool replay and personal-secret setup. Team installation independently passed. The personal native run had actionable final directions, but presentation selected an earlier ordinary comment and suppressed that answer. This pin makes the reviewed corrections available; it does not regrade those attempts or claim a new native pass. ## What Changed - Pin immutable candidate `5f5e2ca8f18063d051c3d948105f0049562fe4ce` on `codex/chat-catalog-four-outcome-repairs-20261104`. - Refresh the existing descriptor snapshot and matching manifest metadata: timeline v5, app-review v5 and personal-secret v4. Company v3 is retained. Preserve the prior reviewed company recipient correction. - Include the reviewed structural standing-chat response selection and finished-turn recovery, preserving earlier comment history, governed interaction ownership and no-final fallback. The superseded prose classifier is absent. - Preserve all 244 identities, original queries, provider/model, human and native counts, clocks, launcher bytes, private evidence limits and qualification holds. The existing SDK registry/compiler proof is unchanged. - The infrastructure change is three metadata files. No product fix is merged by this PR; its owning core PR remains separate. ## Verification - Exact final candidate: 510 owning tests passed across 19 suites, including catalog contract/consistency and lazy declaration loading. The candidate declares 242 cases plus two existing legacy tasks; no fixture callbacks execute during admission. - Candidate response summary suite: 96 tests passed. The owner separately reports 103 summary/notice controls and 14 recovery PostgreSQL controls; these are distinct from candidate results. - Candidate PostgreSQL scope/recovery controls: four passed (three native/ordinary/legacy caller scopes and the new server-finished yielded unknown-channel recovery). The focused filter left 52 unrelated controls unselected. Earlier setup attempts failed before assertions due missing native library links and plugin-sdk output; package-cwd shipped hydration and one owning SDK compiler build enabled this final actual run. No source workaround, timeout change or acceptance weakening was used. - Two deliberate baseline joins preserve the frozen candidate: the caller test adds only its required `issueComments` import; recovery adds only current-conversation matching yielded result/terminal with no owned interaction, retaining all existing succeeded/completed/session/ownership/authorization guards. The test adds only `statusDecisions` and needed fixture options. Fresh-master ordinary `needs_review` behavior and unrelated imports were not copied. - All 131 existing pin, TestOps contract/API, adapter, reconciliation and streaming controls passed. The descriptor snapshot exactly matches the candidate declaration bytes. Diff whitespace checks passed. - SDK source guard: 506 inputs, `changed: false`, `runtimePinsChanged: false`, registry `efa70621e9fb42aa185f7ee616a1a726aa3eccb367e67e28817cedf26ef2ade8`. The local compiler invocation supplied missing ignored test output; it is not a renewed SDK qualification or runtime certificate. - Existing reporting-lock guard returned `already-reviewed`. Fresh protected master `86814c6550f52e99ede150a883bdc773cbb540e3` has admitted lock `60f4c06bca47af3ab9d74281f2ef2e1ff559d566b862ba51c2fbe6c8e5bac35b`. Repeat the existing serving-master guard after merge and before a new Submit. - The earlier pin run exposed a pre-existing remote-failure-retention fixture failure. Normal protected merge #18984 repaired that baseline on master `04a24f4e06da68043378bdf3646f50292efd5316`; its 47 real jobs and aggregates succeeded despite delayed GitHub UI/API status projection. This branch normally merges that actual master. Its three pin metadata files are byte-identical to prior reviewed head `dec78737cc520ce6d489fb8bfccafb460b79e870`, with the same immutable candidate, descriptor bytes and model/count/clock/provenance fields. No baseline fixture patch was duplicated in this PR. - On updated head `bd5402d4688e690ffd9f1f80081577733ceeda25`, the existing 131 controls pass, reporting lock is already reviewed, and existing `selectedCases` admits exactly case 580 without execution. Fresh actual master lock remains admitted. - Exact-head normal CI run 41865757115 has completed its full actual job set: 48 succeeded and one legitimate selector skip, including `ci / e2e` and `ci / verify`. RunsOn completion records and exact GitHub aggregate logs corroborate the results. Windows run 41865757064 completed SUCCESS; its exact log reports all executed Rust test results as passing. GitHub UI/API/check-rollup projections still contain stale queued/in-progress rows, as independently observed during #18984; this is not a claim that those displayed rows are green. A normal protected merge must be accepted by GitHub without override. - Fresh exact-head Greptile is 5/5, the current Paperclip assessment explicitly says 5/5 and Ship, all review threads are resolved (zero threads), and the security scan succeeded. Earlier company-only checks are not used to qualify this head. Full repository build/typecheck/test were not repeated locally for this metadata PR; owning source checks and ordinary hosted CI provide the relevant gates. - No model run, dispatch, retry, cancellation, secret mutation or merge was performed by this update. ## Risks The structural correction changes final-answer selection only for admitted native standing conversations; it still needs a new real secure-entry journey. Oracle corrections can admit previously rejected wording, so retained positive and negative controls remain required. Original failed and unknown attempts are immutable. Company selection remains gated on actual personal-secret secure-entry/worker proof, and all 244 cases remain unqualified as a whole. ## Model Used OpenAI `gpt-6.1-sol` through Codex. Capabilities used: source analysis, isolated integration and local checks. The host does not expose the context-window limit. The evaluated model remains OpenCode/OpenRouter `deepseek/deepseek-v4.1-flash-1017`. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes — manifest and expected-pin assertion agree - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green — actual job/aggregate results passed; stale UI/API projection is disclosed above - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>canary/v2026.1104.0-canary.10 |
||
|
DottaandPaperclip
|
04a24f4e06 |
test(recovery): align retained allocation fixture with readiness (#18984)
## Thinking Path > - Paperclip manages AI agents and their work. > - Restart recovery must preserve the original remote allocation and its files. > - #18934 readies that same retained allocation before preparation. > - The older #18936 failure fixture rejects every readiness call. > - It stops before the intended preparation-failure boundary. > - This change updates only that fixture and checks the original allocation identity. ## Linked Issues or Issue Description **What happened?** The three remote preparation retention cases fail after #18934. Their driver stub throws `Reattachment must not resume the allocation`. The production code now calls readiness on the same validated lease before preparation, so the test never reaches its intended execution failure. **Expected behavior** The fixture should allow readiness of the original allocation. It must reject replacement acquisition and retain its preparation-error, ownership, saved-file and retention checks. **Steps to reproduce** Run `src/__tests__/heartbeat-native-remote-failure-retention.test.ts` on master `86814c6550`. All three cases stop at the stale readiness stub. Refs #18934 and #18936. This one-file baseline correction is split from #18970 so it can unblock the shared failure also observed on #18977. It carries no core API change or evaluation source pin. ## What Changed - Return the original lease from the fixture readiness stub. - Assert exactly one readiness call for the original company, environment, run, workspace, provider allocation and remote path. - Assert that no new allocation is acquired. Keep all existing preparation failure, ownership and retained-work assertions. ## Verification - Before #18934, the three existing cases passed locally. On the actual merged #18934 contract, they reproduced the three CI failures in 18.06 seconds. - With this fixture correction, all three passed in 11.83 seconds. Production runtime code and the original 60-second case deadline are unchanged. - The corrected test blob and its relevant production/helper blobs match the already-tested merged-source proof. Current master still has the same stale test blob; this isolated PR applies only the identical fixture correction. - Command from `server/`: `node ../node_modules/vitest/vitest.mjs run --config vitest.config.ts src/__tests__/heartbeat-native-remote-failure-retention.test.ts`. - `git diff --check` passes. Full repository typecheck/build were not rerun for this one-file fixture split; fresh normal CI and review are pending. - [Original failing CI job](https://github.com/paperclipai/paperclip/actions/runs/41855440307/job/126665077718). It remains failed; no remote retry was used as proof. ## Risks - Low scope: test fixture only. A readiness call for a replacement or wrong allocation fails the identity assertions. - This does not alter production recovery, permissions, cleanup policy, evaluation deadlines or grading. ## Model Used OpenAI `gpt-6.1-sol`, with reasoning, repository inspection, code editing and tool execution. The host did not expose the model context-window limit. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
86814c6550 |
fix(recovery): ready retained sandboxes before preparation (#18934)
## Thinking Path > - Paperclip manages work performed by AI agents. > - Restart recovery must preserve each run's workspace and execution owner. > - A retained remote lease can refer to an archived sandbox. > - The reattachment path returned that lease without making it ready. > - A short project-authentication command then had to start the sandbox. > - This PR resumes and verifies the exact retained allocation before preparation. > - Recovery keeps the recorded workspace and preserves saved work if resume fails. ## Linked Issues or Issue Description **What happened?** A native run resumed after a controller restart. Its saved sandbox had been archived. Reattachment returned its lease without a provider lifecycle resume. The next project-authentication probe tried to start that sandbox with its 15-second command budget. The start failed before the probe could inspect project settings. The heartbeat reported a configuration conflict and stopped work. **What did you expect to happen?** Make the exact retained sandbox ready through the existing bounded lifecycle operation. Verify its allocation before running short preparation commands. Keep the validated lease and recorded working directory. Never acquire a replacement in this path. **Steps to reproduce** Hold the provider's exact-lease resume pending and call remote reattachment for an active ephemeral lease. Before this change, acquisition returns immediately. The added tests hold acquisition until resume completes and reject missing, changed or expired allocation evidence. Provider compatibility cases retain the recorded workspace when resume omits directory metadata or returns the allocation root. **Deployment mode** Native Runner in a sandbox with a retained workspace and managed AI connection. The observed provider was Daytona. Refs #18910, which restores selection of retries using current remote workspace storage. This change repairs the next preparation boundary. Related #17421 and #18559 change project-file inspection; they do not make retained sandbox leases ready. ## What Changed - Resume the exact validated sandbox lease before returning it to preparation. - Require the receipt to confirm the original provider allocation without reporting expiration. - Keep the original database lease and recorded directory. Resume metadata can describe the allocation root. - Pass the existing typed resume-directory input through the runtime facade. - Preserve retained E2B sandboxes when resume timeout setup or workspace inspection fails. Keep new-allocation cleanup and explicit release/destroy behavior. - Preserve plugin-task reattachment and lease identity checks. - Update the Stop-accounting fixture to confirm exact-allocation resume before cleanup, with its budget, ownership and no-paid-work assertions intact. - Document the ordering and add readiness, receipt, failure and ownership regressions. ## Verification - Red: seven initial retained-lease cases failed before the readiness change. Two receipt-compatibility cases and four E2B preservation cases failed before their fixes. - Green: all four runtime, workspace-preparation, heartbeat-recovery and orchestrator suites pass: **442 tests**. - The Stop-accounting CI failure reproduced locally. All **five** integration cases pass with the exact-allocation resume fixture. - All **22 E2B tests**, the standalone plugin typecheck and its build pass. - **13 Daytona safety cases** pass, including recorded directories, allocation sentinel rejection, saved-file reads and no replacement creation. The SDK boundary is simulated; the saved-file case uses real files and a POSIX shell. - These focused checks cover **482 passing cases**. - Full recursive typecheck, final server typecheck and full workspace build pass on macOS arm64 with Node 24.22.2 and PostgreSQL 18.1. - Greptile is **5/5** on head `32f915a475275680bcf2a2aa43ae0c72f4e4b1db`. Both original review threads are resolved. - All current-head CI gates pass. The check rollup has 54 successful checks, three skipped checks and one neutral check. - The separate Paperclip Review check remains queued. Its public response says work is paused and the request is saved. The review task is waiting for saved-file recovery, so no Paperclip assessment of this head is claimed. The branch-required gate has passed. - A read-only deployed trace showed an archived sandbox start fail at the short probe's 15-second deadline. Actual provider reattachment with this revision has not been exercised. Live recovery remains unverified until rollout. ## Risks Recovery now makes one bounded lifecycle call before short preparation commands. A missing resume capability or a missing, changed or expired allocation receipt holds recovery. The existing runtime and provider retain allocation and sentinel verification. The path does not acquire, release, replace or reseed the lease. Warm providers can return their existing allocation. E2B retains the existing sandbox if resume setup fails; explicit lifecycle cleanup still owns its removal. No background polling, credential-policy change, migration, workflow change or lockfile change is added. ## Model Used OpenAI GPT-6 through Codex, with reasoning, source inspection, tool use and code execution. The exact backend snapshot and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
32e5eada06 |
test(evals): pin reviewed attribution and secure question repairs (#18974)
## Thinking Path > - Paperclip must complete work through ordinary human and agent controls. > - The remote chat campaign exposed observer and setup defects as well as product discovery gaps. > - Case 466 rejected a valid handoff that used headings, pronouns and inline HTML. > - Case 487 seeded three daily runs although its declared journey needs six. > - Secure-entry replies used a real bound question card that the observer did not support. > - Team installation omitted its existing strict request schemas from the API search result. > - This PR pins the reviewed coherent repairs and keeps their metadata in sync. ## Linked Issues or Issue Description Refs #18921. Refs #18964. Refs #18967. Refs #18968. Refs #18970. Refs #18971. Refs #18972. The previous launch remains unchanged. Case 53 passed its latest native attempt. Cases 466, 487, 580 and 610 still retain their prior failures. The next representative run must test the repaired conditions. The company-secret case follows only after the personal-secret journey proves real secure entry and worker use. Its separate case-local follow-up is held for a later pin; case 578 is excluded from the next four-case selection and has no pass credit. ## What Changed - Pin immutable candidate `e757e447371c4781a3eff3a0ea9dc9c4eceb060a` and its existing descriptor snapshot. - Record the explicit oracle versions and source-file changes for timeline attribution, review fixture allowance, and bound personal/company secret questions. - Keep all 244 case identities, human requests, original run and action counts, time allowances, provider/model, private evidence limits and broader qualification holds unchanged. - Preserve the actual request-body schema affordance for team preview and installation, existing response links and prior help. ## Verification - The final integrated candidate passed 442 owning checks across 18 suites. All 242 declarations loaded with zero fixture callbacks; the existing manifest joins those declarations and two legacy tasks into 244 cases. - The team service fixture passed 29 tests. The frozen candidate API consumer passed 2045 tests after the three-line schema exposure change. These are component controls, not a new native pass. - The stale inventory expectation reproduced one failure with 16 passes. Its unchanged derivation traces five previously reviewed YAML corrections. Updating only the expected constant passed 17 controls, and the existing definition-drift and catalog-tamper negatives still passed. No scenario YAML changed in this candidate. - The existing SDK guard confirms all 506 inputs are unchanged and the reviewed registry remains `efa70621e9fb42aa185f7ee616a1a726aa3eccb367e67e28817cedf26ef2ade8`. No compiler build or registry renewal was needed. - The existing pin, TestOps contract, adapter, reconciliation and streaming controls passed locally. Current master also passed the existing reporting-lock guard with its actual current lock. - The final source includes the reviewed immediate-actor/consecutive-pronoun controls and checks every rendered question title, header and submit label for unsafe instructions. The remaining fixed scanner reasons are diagnostic only. The combined source passed 442 checks; all 131 pin/contract/API/streaming controls passed. - Fresh exact-head [normal CI](https://github.com/paperclipai/paperclip/actions/runs/41847423336) passed: 48 successful jobs and one selector skip, including both required aggregates. Greptile reports 5/5 for this head, with no actionable findings or unresolved review threads. The current optional Paperclip review is queued and is not credited as an assessment pass; its prior-head 5/5 remains historical. The finite source peer review found no material issue and did not claim native execution. The prior CI attempt is retained. No new native pass is claimed by this PR. ## Risks The new source repairs observation and setup, but it does not prove the next native outcome. The secure-entry accounting records one real chat request and one authenticated UI answer as the two declared human actions; it keeps the raw comment count separate. The question path requires the bound pending card, actual resolver audit and delivery facts. Ordinary cases retain their existing accounting. Private scans and archive limits can still reject evidence. Prior failed and unknown attempts remain immutable. ## Model Used OpenAI `gpt-6.1-sol` through Codex. Capabilities used: source analysis, integration and local checks. The host does not expose the context-window limit. The evaluated model stays OpenCode/OpenRouter `deepseek/deepseek-v4.1-flash-1017`. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
84e9c8f364 |
fix(retry): wait for admitted source cleanup before dispatch (#18898)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Connected-task Retry keeps the original request and its current authority. > - One durable Retry can already own a queued successor. > - The old native environment can still have pending cleanup before that successor starts. > - The task execution lock points to the successor, so the old cleanup can escape the queue claim check. > - Dispatch then refuses the source state and fails the new run before provider work. > - This pull request keeps that same run queued and wakes dispatch when cleanup changes. ## Linked Issues or Issue Description Refs #18845, #18853, #18860, #18877, #18882 and #18932. These repairs cover preservation binding, interrupted preparation, old native ownership, manual native holds or other setup guards. This change covers an already admitted connected-task Retry whose native predecessor still owes cleanup. It uses the shared cleanup predicate so the manual-hold repair remains with its existing owner. Search found no duplicate queued-source cleanup repair. **What happened?** A board Retry saved its exact chat request and admitted one queued successor. The source lease was still pending cleanup when that successor reached dispatch. A retained stop receipt did not settle the full cleanup contract. The queue checked the task's current execution owner, which was already the successor. The native authorization guard then rejected the predecessor state and failed the new run before provider dispatch. **Expected behavior** Keep the same canonical Retry and queued run while the exact source cleanup is pending. After authoritative settlement, submit that run through normal dispatch once. Do not infer safe execution from a stop-and-retain receipt alone. **Steps to reproduce** 1. Retry an authorized connected request from a failed native run. 2. Admit one successor while capacity keeps it queued. 3. Leave or restore the source lease to pending cleanup with a matching retained stop receipt. 4. Release capacity and resume the ordinary queue. 5. The old claim path starts the successor and the native source guard refuses it. **Paperclip version or commit** Reproduced against master at `f77499283270d2e966aabd58493f9d35c6bbdf5f`. **Deployment mode** The regression uses the real HTTP route, chat authorization, queue claims, disposable PostgreSQL and the production work coordinator on macOS arm64. A controlled adapter supplies the final result. No hosted provider replay or deployment is claimed. ## What Changed - Validate the exact canonical Retry, chat action, company, task, agent, source and successor before testing source cleanup. - Keep the successor queued under the existing issue claim lock while the shared cleanup predicate remains pending. - Wake only the scoped queued-dispatch worker after cleanup or exact workspace-owner release commits. Add no timer or background scan. - Verify pending cleanup at intake, later cleanup invalidation, one final execution, idempotency, Stop, changed input, unknown stop receipts, a different live controller and signed sandbox absence followed by owner release. - Document the queued-wait contract. No schema, endpoint, credential, model or provider policy changes are required. - Align five native preparation acceptance fixtures with the existing workspace publication contract. Use the real reservation and publication owner. Retain a raw-root refusal case with no provider launch and preserved files. ## Verification - Latest master sync at `e26cad76b8f20ba823d702ed60e7b7294f8f5a7c` includes master `396b7fc62a0a4b1d8aadbda8221c05879d54177c`. The shared test helper preserves #18944's canonical retained-directory default, the real publisher for the five fresh-workspace cases, and the explicit raw-root refusal. Independent remerge review found no dropped upstream behavior, guard or assertion. - The latest head passes the focused six cases, all 580 complete workspace/continuation tests, all 38 selected HTTP Retry cases, fresh full recursive Node 24.22.2 types and full build. No affected tests skipped. An initial cached test-layer attempt read old budget declarations; regenerating them resolved all seven errors without test/mock or production-export changes. - All current-head CI is green at `e26cad76b8f20ba823d702ed60e7b7294f8f5a7c`, including required [ci / verify](https://github.com/paperclipai/paperclip/actions/runs/41846304017/job/126636751314) and [ci / e2e](https://github.com/paperclipai/paperclip/actions/runs/41846304017/job/126636638086). Greptile and Paperclip Review both rate this exact head 5/5, with no findings or open review threads. - Master sync at `3e4f266c2da172a84afd0b9cb4a91ea3b95f3c0d` merges master `b891e69bd23d6dab8ce536569e15bc4a7a31c35c`. The only conflicts were adjacent queue imports and two HTTP Retry test matrices. Both imports and both complete matrices remain unchanged. Independent review found no relaxed guard or assertion. - Node 24.22.2 and pnpm 9.15.4: all 38 selected real HTTP Retry cases, 194 queue/cleanup/finalizer cases, module boundaries, recursive types and build pass at `3e4f266c2da172a84afd0b9cb4a91ea3b95f3c0d`. - That sync head's [server check](https://github.com/paperclipai/paperclip/actions/runs/41838741852/job/126604742761) failed five native acceptance cases in CI and locally with `remote_workspace_published_path_invalid`. The test fixture supplied a raw directory instead of a fresh publisher-generated workspace. The guard from #18932 remains unchanged. The same five cases passed earlier Linux CI before that guard was present. - The test-only alignment at `dcdee1bb6c2e2e6dd494382499d17c23a3ce71a8` obtains the saved authority, path and run reference from real `reserveRemoteWorkspace` and `prepareRemoteWorkspace`. All five original cases and a raw-root no-launch/no-result/files-preserved control pass. Their native result, file, catalog and single-execution assertions remain intact. An initial setup attempt was refused for a noncanonical allocation root; the fixture now resolves that owned root before publication. - At previous fixture head `dcdee1bb6c2e2e6dd494382499d17c23a3ce71a8`, the generated server test-layer typecheck passed. All 580 tests in the complete workspace/continuation family pass, with no skips. That test-only commit left production files byte-identical to the sync head. - Previous published-head CI was green at `dcdee1bb6c2e2e6dd494382499d17c23a3ce71a8`, including required `ci / verify` and `ci / e2e`. The [Linux Node 24.22.2 server shard](https://github.com/paperclipai/paperclip/actions/runs/41842665765/job/126618334638) executed all five original acceptance cases and the new raw-root refusal case successfully. Its whole workspace recovery file passed all 245 tests. - Greptile reviewed previous published head `dcdee1bb6c2e2e6dd494382499d17c23a3ce71a8` at 5/5, with no findings or open review threads. No unchanged-job rerun was used for the reproduced sync-head failure. - Causal red: the real HTTP regression started and failed the successor at the native source-state guard while cleanup remained pending. A second red proved that signed sandbox absence followed by copyback-owner release lacked a queued-dispatch wake. - Green: seven HTTP/PostgreSQL cleanup cases, the exact confirmed-owner release notification control and seven existing retry lineage controls pass. The real coordinator resumes from committed cleanup or workspace-owner release, without a manual queue restart. One controlled adapter execution produces the result after settlement. - Earlier branch verification passed full repository `pnpm -r typecheck`, `pnpm build`, module boundaries and `git diff --check`. - The full stale-queue file control passed all 64 cases with unchanged observer/hook limits. The initial broader affected run had 192 passes and one non-assignee native-owner case miss its existing five-second adapter observer while already running with no error. The isolated case passed; the unchanged base passed all 64 cases but hit its existing 30-second teardown timeout. Those original results are retained. The new queue hint has no dispatch consumer in that fixture. - Earlier-head CI was green: 54 success, three skipped and two neutral checks at `6adb9f1ba682282c512d5e751ba2aa663f614ca2`. One chat job was cancelled by an external runner shutdown; its single unchanged-job rerun passed. No source was changed to obtain that result. - Greptile and Paperclip Review both rated earlier head `6adb9f1ba682282c512d5e751ba2aa663f614ca2` 5/5 with no findings or unresolved threads. These results do not qualify the master-sync head. - At initial verification, a complete local `pnpm test:run` was not repeated while the related setup repair's full local run was active. The full earlier-head CI suites passed. A full local suite has not been repeated for the mechanical master sync. Local verification uses the affected HTTP, producer, lineage and queue suites. No full local-suite pass is claimed. - The fresh clone's missing plugin SDK links were repaired with the repository's own postinstall helper before verification. The embedded PostgreSQL package's own symlink helper was also required. - No staging task was submitted again with this repair, and no deployment is claimed. ## Risks - A pending source cleanup can keep the queued run waiting. Its cleanup owner still owns reconciliation and its existing limits. - Stop, changed source, unknown termination and live ownership retain their existing authority checks. A cleanup notification grants no execution authority. - A new cleanup change after claim still faces the existing final dispatch guard. - Cleanup notifications can change dispatch timing in a running controller. The real coordinator cases verify commit-bound wakeups; the broader current queue-file control passed all 64 cases. The earlier loaded-run and baseline teardown failures are retained in verification. ## Model Used OpenAI GPT-6 through Codex, with reasoning, code editing, tool use, PostgreSQL and browser verification. This session does not expose a more specific deployment model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
5d226ca08a |
perf(chat): reuse bound-run authorization query templates (#18966)
This PR depends on #18969. Merge that fixture-only prerequisite first, then retarget this PR to master. No merge or deployment is requested in this work. Its base is `codex/fix-recovery-published-workspace-fixture`; the six-line recovery fixture repair is excluded from this diff. ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - External chat authorization binds each run to current messages and conversation policy. > - Review wait and attention-retention paths repeat the same four-table read. > - Each call rebuilt the Drizzle query even when its SQL shape stayed the same. > - This pull request reuses the compiled template for the exact database handle and lock mode. > - Fresh parameters and current database reads preserve authorization while reducing Node CPU work. ## Linked Issues or Issue Description **What existing behavior does this improve?** The `authorizeChatConversationForBoundRun` links read used by external chat waits and attachment reuse. **Current behavior** At baseline `e54f9fbf828e06e385e0c62dc381ccb46190aa30`, each call rebuilt and serialized a four-table query. The observed staging shape accounted for 16.2% of sampled SQL serialization CPU across 32 samples. This is a share of query compilation CPU, not total server CPU. **Proposed behavior** Compile each read, blocking-lock or NOWAIT variant once per exact live database or transaction object. Bind new company, issue, agent, provider and comment IDs on every execution. Continue all current database reads and authorization checks. **Reason and benefit** Repeated authorized service calls used 26.91% less Node process CPU in the matched local fixture. All seven CPU pairs improved. This reduces repeated compilation on the shared service path. Related: #18896 retains task Retry intent through identity contention. It changes a different workflow and does not reuse this query template. A public issue and PR search found no duplicate template change. ROADMAP.md has no conflicting planned feature. ## What Changed - Keep compiled queries in a `WeakMap` keyed by the exact supplied database or transaction object. Create only the requested lock variant. Never cache a result or authority decision. - Bind the comment list as `ANY(uuid[])`. This supports changing list sizes with only three templates per live handle. Keep all joins, selected fields, policy checks and lock clauses. - Add real PostgreSQL regression tests, a reusable fixture, an actual service benchmark and a dated performance note. ## Verification - Passed 17 real PostgreSQL tests: `node node_modules/vitest/vitest.mjs run --project @paperclipai/server server/src/__tests__/prepared-chat-authorization.integration.test.ts server/src/services/native-runtime/chat-attachment-reuse.test.ts`. - After rebasing, seven selected existing external-chat-wait integration cases passed (266 deselected). They cover small current review waits, answered-chat identity advisory locks, connection-disable serialization and contended endpoint proof retry. - New tests cover changed parameters and list sizes, duplicate/empty inputs, invalid UUID errors, company/agent/provider binding, current setup generation and credential fingerprints, root/transaction/savepoint isolation, rollback, read mode during a held row lock, blocking timeout and NOWAIT contention. - Actual service benchmark: `node --import ./server/node_modules/tsx/dist/loader.mjs server/scripts/benchmark-bound-chat-authorization.ts --baseline-ref=e54f9fbf828e06e385e0c62dc381ccb46190aa30`. Node 24.22.2, macOS arm64, real PostgreSQL, 500 warmup calls per implementation, seven alternating pairs of 500 calls. Every result matched. - Pinned measured treatment `8386eb78d118faaf6c2fd7f7471b38c0f4b2ef54`. Its source is unchanged after rebasing onto master. Median Node CPU per 500 calls: 379.033 ms to 277.048 ms, down 26.91%. Every CPU pair improved by 13.54–33.52%. Median wall time: 742.109 ms to 648.005 ms. One wall pair increased by 0.51%. - A separate diagnostic after timing found five SQL compilations per baseline call and four per warm treatment call. All five database reads still run. - The final four-test PostgreSQL authorization suite passed again after rebasing and strengthening the lock proof. It holds only a message-link row, so principal advisory contention cannot mask a missing joined-query lock. - Independent source review found no correctness issue in exact-handle ownership, fresh binding or the three lock variants. - Preserve setup failures: the first test attempt lacked plugin SDK build output. Shared and SDK builds fixed it. Later attempts could not start embedded PostgreSQL because its local dependency directory lacked bundled native-library aliases. `DYLD_LIBRARY_PATH` did not fix it. Restoring package-local aliases fixed the environment; the final 17 tests then passed. No dependency or lockfile change is included. - The first full typecheck found new test-only errors: PostgreSQL transaction handles lack the root database `$client` type, and runtime setup test data includes fields outside the saved setup interface. Commit `40a4645750` uses the existing explicit transaction test cast and SQL JSON fixtures. No production contract changed. The full check also found local plugin SDK link prerequisites, which the coordinator is repairing separately. - Preserve the first Linux CI failure: [server shard 9](https://github.com/paperclipai/paperclip/actions/runs/41837775564/job/126602280304) failed the new setup-fence fixture's allowed case. The fixture supplied a client `processedAt` while PostgreSQL supplied default `receivedAt`; this could violate the required processing order. The test now uses explicit ordered timestamps and no wall-clock window. The four-test PostgreSQL suite passed again in 19.93 seconds. Production guards were not weakened. - [Server shard 2](https://github.com/paperclipai/paperclip/actions/runs/41837775564/job/126602280291) also failed five workspace-recovery cases with `remote_workspace_published_path_invalid`. Those failures reproduced on both pure master and the exact CI merge with the chat optimization removed. #18969 fixes the canonical path in that separate fixture, with five failing cases before and five passing after. This PR is stacked on that independent repair for final CI. - The additional Paperclip review reported a node-postgres statement-name collision and requested explicit pool cleanup. This repository uses postgres-js; locked Drizzle 0.45.2 accepts the Drizzle label but does not use or forward it. [The review response](https://github.com/paperclipai/paperclip/pull/18966#issuecomment-6632666599) links primary driver source and the database factory. Commit `c5d0d84f36` adds explicit pool teardown in `try/finally` and a real same-transaction `read → blocking → nonblocking → read` case. All five authorization tests passed in 6.47 seconds. The original separate row-lock test remains. Production SQL and code did not change. - Final combined local validation at `4ab58ff70b` passed full `pnpm -r typecheck` (39.864 seconds) and 35 real PostgreSQL cases across the chat, GitHub and shared prepared-query suites (11.378 seconds). Full `pnpm build` passed earlier with byte-identical production source; later changes affect tests only. The production file SHA matches the benchmark and this PR. The original full `pnpm test:run` was interrupted after all six chat shards passed and general tests had partly run, so it is not a full local pass. Final current-head CI provides the broader coverage. A sandbox retry failed with EPERM before assertions; the identical escalated commands then passed. Final head passed CI in [workflow 41843806060](https://github.com/paperclipai/paperclip/actions/runs/41843806060), including all build, typecheck, general, serialized and e2e test gates. The previous head `40a464575009dfc3cce7afa1eb4fffa5540dcaf6` passed Greptile 5/5, security review, typecheck, build and canary validation. The prior stacked head `74cb36370162750f8e135ed091603d3629d23b0d` passed full CI, Greptile 5/5, policy and security checks. Final head `c5d0d84f369d897a66d9b6441bab10cb79f593b9` passed a fresh Greptile 5/5 review with no actionable findings, security and PR policy checks. Its complete [current-head CI workflow passed](https://github.com/paperclipai/paperclip/actions/runs/41843806060). All 54 required CI, Greptile, policy and security checks passed. The optional Paperclip Review check is queued for this head; no current-head score is claimed. Production source SHA-256 remains `72ce9d2317e5c97b36e299e24e30fc1aa63020d8c266e495fc45dbfdd5331e93`, identical to the measured code. ## Risks - `ANY(uuid[])` relies on postgres-js array binding and UUID casting. Real PostgreSQL tests cover single and multiple valid values and invalid UUID errors. - A new transaction or savepoint pays its first compilation cost. The weak cache cannot reuse a root session for it. This benchmark covers repeated calls on one root handle. - The team paused other work during the benchmark. Unrelated Rust compilation was active elsewhere on the host. Process CPU excludes those processes, but host contention can affect caches, scheduling and wall time. This source-mode local result does not prove production request latency or whole-server CPU savings. - No migration or API contract changes. No deployment or merge is included. Monitor authorization failures and lock-contention behavior after an authorized rollout; revert the narrow service change if those regress. ## Model Used OpenAI `gpt-6.1-sol` through Codex, with reasoning, repository editing, tool use and test execution. The host did not expose a context-window size or deployment snapshot ID. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
DottaandPaperclip
|
2cbfc35a07 |
perf(github): reuse the inherited review wait query template (#18965)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - External review waits check current GitHub review and publication bindings. > - The GitHub service rebuilds the same seven-table Drizzle query on each check. > - A sampled production window attributed 6.3% of SQL serialization CPU to this query shape. > - This pull request prepares that query once per exact database or transaction object. > - Each call still reads current provenance, review, connection and publication state. > - The local service benchmark reduces median client CPU per read by 24.35%. ## Linked Issues or Issue Description **What existing behavior does this improve?** Repeated `readInheritedGitHubReviewWait` calls from external review wait reads and attention retention. **Subsystem affected** Server GitHub chat review waits. **Current behavior** Every call constructs and serializes the same joined SELECT. The database driver already supports prepared statements. That setting does not reuse the JavaScript query template. **Proposed behavior** Compile the existing selector once for each supplied database, transaction or savepoint object. Bind company, issue, agent and comment IDs on each execution. Keep every current database read and check. **Reason and benefit** Remove repeated query construction and serialization CPU. Do not cache rows or authorization decisions. Exact open-PR searches found no duplicate query-template change. ## What Changed - Add an exact-handle `WeakMap` for the GitHub review selector and execute it with fresh placeholders. - Preserve the SQL projection, joins, JSON types, latest-review order and all current authority checks. - Extend real PostgreSQL tests for new bindings, warm-cache revocation, latest heads and transaction/savepoint rollback. - Capture prepared executions in the existing test helper. - Add an actual-service benchmark and a report with every timing sample and setup failure. ## Verification - `github-review-wait.test.ts`: 16 tests passed on real PostgreSQL. - `external-chat-wait.integration.test.ts -t 'reads small current external review waits without saved prompts'`: four affected cases passed; 269 unrelated cases skipped. - Pinned baseline: `e54f9fbf828e06e385e0c62dc381ccb46190aa30`. Final source SHA256: `fe647e7fe2aed82609e8f1d66a0109e7a10ee28313ae2ef218ba3ab39b5cd105`. - Exclusive-window benchmark: 500 warmups per version, nine alternating pairs of 100 complete service reads on disposable PostgreSQL. Median client CPU: 1.00336 → 0.75908 ms/read (24.35% less). Median wall time: 2.87089 → 2.38731 ms/read (16.84% less). CPU excludes the PostgreSQL process. - Every generated SQL statement, parameter, query count and complete result matched. Warm joined-selector compilations per ten service reads: 10 → 0. - See `doc/performance/2026-11-04-prepared-github-review-wait.md` for reproduction, all samples and measurement limits. - Setup record: loopback required execution outside the sandbox. The fresh embedded-Postgres dependency lacked macOS dylib aliases; aliases to existing identical bytes restored the fixture. The SDK import needed a current workspace SDK/shared build. A broad external-chat suite run was interrupted and replaced by the affected test filter. Those attempts are not counted as passing suites. - Greptile: 5/5 on current head `d2e6ae8814c3a56c380bd5d60c47e6f98642f085`, with no actionable issues or inline comments. - Current-head CI is green at `d2e6ae8814c3a56c380bd5d60c47e6f98642f085`: 52 successful checks, three intentional skips and the successful Snyk status. This includes typecheck, build, the full test matrix, end-to-end shards and the aggregate gate. - Combined local validation at `4ab58ff70b` includes both measured production templates and final test fixes. Full recursive `pnpm -r typecheck` passed in 39.864 seconds. Three real PostgreSQL files passed 35/35 tests (five chat cases, 16 GitHub review cases and 14 shared prepared-query cases). The combined build passed earlier with the same production template bytes. - The initial combined typecheck failed on a separate chat test typing error and build setup. The later passing run resolves that attempt. The original full local `pnpm test:run` was deliberately interrupted after six chat shards passed and partial general tests ran; it is not a passing full local suite. Each PR has independent current-head CI for broad coverage. The initial loopback sandbox restriction was resolved by an escalated run of the same source. - Local benchmark/test dependencies were locked at the baseline; the rebase adds the upstream lockfile refresh and UI paste fix only. ## Risks - A prepared query captures its session. The exact supplied handle is the cache key so transactions and savepoints do not borrow the root session. - Placeholder mistakes could cross bindings. Tests alternate independent company/issue/agent/comment fixtures on the same handle. - The benchmark has one synthetic eligible fixture with warm sequential reads. It does not predict production endpoint latency or whole-server CPU savings. - No schema, index, database driver setting or public API changes. ## Model Used OpenAI `gpt-6.1-sol` through Codex. The context-window size is not exposed by the session. Used reasoning, tool calls, source editing and code execution. Independent Codex review found no correctness issues. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |