Commit Graph
7726 Commits
Author SHA1 Message Date
DottaandPaperclip c6cc59282a fix(evals): separate dash-prefixed credential requests
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-04 15:32:41 -06:00
DottaandPaperclip 4740d251b7 fix(onboarding): retain plan approval across follow-up decisions (#19043)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Onboarding asks a person to approve a saved plan before execution
tasks can start.
> - A person can then approve a separate proposal for the first
execution step.
> - The creation gate checks only that latest card for the plan target.
It loses the earlier approval of the unchanged plan.
> - Decomposition and native context also omit checkbox plan
confirmations.
> - This pull request validates the latest execution decision and the
latest plan decision together.
> - The benefit is that approved work can start while human, revision,
and proposal gates remain effective.

## Linked Issues or Issue Description

Refs #17914, #18160, and #18283. The related-PR search found no
duplicate repair.

**What happened?**

After a person accepted a checkbox plan card and a later execution
confirmation, native task creation and REST child creation returned 403.
Accepted-plan decomposition returned 422 because its lookup recognized
only standard confirmations. Native task context returned no accepted
plan for the checkbox decision.

**Expected behavior**

The latest execution proposal can use an independently approved,
unchanged plan. A pending or rejected latest proposal must still stop
execution. The current plan must retain valid human approval of its
exact revision.

**Steps to reproduce**

1. Save a plan on a server-created onboarding task.
2. Create a revision-bound checkbox confirmation and accept it as a
human.
3. Create and accept a separate, untargeted confirmation for the first
execution task.
4. Create the child through the native tool or REST API.
5. Separately, try accepted-plan decomposition or read native task
context after step 2.

**Paperclip version or commit**

The workspace's registered hosted release is Core
`09c05c55050fede448196637cbaa543a154da76e`. The three regressions also
failed against local source `3cdb367369f29692b81cf19b514a2abdba90734f`.

**Deployment mode**

Hosted native Runner with ACPX Claude. Local regression proof uses
actual PostgreSQL, authenticated HTTP routes, native tool authority, and
synthetic task data on macOS arm64. No customer data or credentials are
in the fixtures.

## What Changed

- Validate the newest ordinary proposal and the newest plan proposal
separately. Keep the existing human-card and conversational provenance
checks for each.
- Require the accepted plan target to match the current document and
revision. A newer pending, rejected, invalid, or foreign plan decision
cannot fall back to an older approval.
- Share persisted confirmation parsing across admission, decomposition,
and native context. Validate checkbox selections against the saved
options and limits.
- Return `acceptedPlan.approvedOptions` with the user's saved selected
IDs and option text on later wakes, including revision-pinned context.
Keep unchecked defaults out of scope.
- Add behavioral and safety regressions. Document the approval rule.

## Verification

- Three behavioral regressions failed before the repair: child creation
returned 403, decomposition returned 422, and native context returned
`acceptedPlan: null`. All three now pass.
- Two scope-retention regressions failed before the review fix: native
context omitted the selected option after a later approval, with and
without a pinned plan revision. Both now pass. Unchecked defaults stay
excluded.
- Current admission, native-tool, and accepted-confirmation replay
suites: 131 tests passed. Existing first-task completion safety checks:
17 passed, including the atomic decision recheck.
- Safety cases cover missing and rejected plan approval, agent-only
decisions, foreign source runs, changed revisions, invalid checkbox
selections, newer pending or rejected proposals, and task-creation
replay.
- Full repository `pnpm -r typecheck` and `pnpm build` passed locally on
final head `7c0e8ab784`.
- [Full final-head
CI](https://github.com/paperclipai/paperclip/actions/runs/41911580531)
passed: all Vitest shards, Runner checks, eight browser E2E shards,
build, release registry, and canary dry run. The required `verify` and
`e2e` aggregates passed. Storybook checks were skipped for this backend
change; the file-selection job was skipped because full CI ran.
- Earlier local typecheck attempts exposed Copilot verification during
build preparation, stale declarations across the rebase, and two
test-fixture typing errors. The assets were verified, declarations
rebuilt, and fixtures corrected before the complete final-head pass.
- Full local Vitest attempts were interrupted for the rebase and review
repair. The duplicate final local run was stopped after full Linux CI
passed. No complete macOS Vitest-suite pass is claimed.
- A fresh Claude-provider acceptance run remains unverified: the local
Claude account is signed out and no Anthropic API credential is
available. The actual native tool and HTTP paths are verified with real
PostgreSQL persistence. This PR does not deploy or recover the reported
production task.
- Greptile: 5/5 on `7c0e8ab784`, with the review thread resolved.

## Risks

A later acceptance must not revive rejected or changed scope. Admission
validates both current decisions and retains source, company, human
provenance, and document locks. The extra plan-proposal lookup is
bounded and limited to accepted, untargeted first-task proposals.
Ordinary delegation adds no new proof lookup. The native context adds an
optional `approvedOptions` projection. Revision-pinned context now
performs the existing scoped approval read to recover that selection. No
database schema, request, provider, or background worker changes are
included.

## Model Used

OpenAI Codex, exact model `gpt-6.1-sol`, max reasoning effort, with
repository, shell, browser, and GitHub tools. The current session
metadata confirms the model and effort. The context window size is not
exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-04 15:24:58 -06:00
Dotta 8002dfb06e fix(evals): repair bounded TestOps protocol admission (#19011)
## Thinking Path

> - Paperclip lets people manage AI agent work through a governed
control plane.
> - Runner protocol qualification uses real providers and records
selected cases in TestOps.
> - A manual registration run starts an Allure watch session before it
validates the case selection.
> - Watch creates the job run, but it does not fill the empty job-run
input in the child environment.
> - The selection helper therefore stops before it can validate the new
job run.
> - A configured AWS concurrency ceiling can also reject a valid smaller
hosted-runner request.
> - Hosted tool-cache permissions can also violate Runner
qualified-executable admission before a provider starts.
> - This pull request captures official session identities, keeps both
concurrency ceilings, and stages the same Node bytes in an owned
directory.

## Linked Issues or Issue Description

Refs #18985.

**What happened?**

The [first registration
attempt](https://github.com/paperclipai/paperclip/actions/runs/41877603952)
created an Allure job run, then stopped with
`runner_testops_jobrun_missing`. The child inherited an empty
`ALLURE_JOB_RUN_ID`. The helper made no job-run API read and executed no
provider case.

The current configured concurrency ceiling is 100. The controlled hosted
cap is 57. The original catalog guard rejects 100 before it applies a
valid requested value of 2. The [first service-selected
attempt](https://github.com/paperclipai/paperclip/actions/runs/41884365232)
reproduced this second pre-provider failure. Its selected case was
correctly reported as broken. No provider cell ran.

A [second service-selected
attempt](https://github.com/paperclipai/paperclip/actions/runs/41887671585)
passed selection, lock and cost preflight. Runner rejected a qualified
ACPX launch artifact as group- or world-writable. It recorded no model
request. The verified sidecar is mode 0755. Hosted Node is the only
other launch artifact, which points to Node as the cause. The original
run did not retain its exact mode. The TestOps reporter retained the
infrastructure failure.

**Expected behavior**

The manual path must capture the job-run and launch IDs from the active
official watch session. It must validate those IDs against the same
GitHub run, job, launch and standing plan before provider work. A
requested concurrency value must fit the configured ceiling and the
selected runner cap. The default must use the lower ceiling. Before
credentials, the worker must stage the same Node bytes in an owned
directory and preserve Runner permission and digest admission.

**Steps to reproduce**

1. Dispatch the trusted Runner protocol workflow with reporting enabled
and an empty job-run service input.
2. Let `allurectl watch` create the registration session.
3. Observe that the original helper rejects the empty child input before
selection validation.

## What Changed

- Call the supported `allurectl job-run env` command for manual
registration. Use a fixed argument list without a shell and bound its
time and output.
- Parse only numeric job-run and launch IDs. Reject missing, duplicate,
malformed or conflicting values. Discard all other command output.
- Bind the captured launch ID to the authoritative job-run response.
Keep the service-supplied ID path and its existing checks.
- Apply requested parallelism within both configured and physical
ceilings. Use their minimum as the default. Keep the selected-case
capacity and provider budget checks.
- Snapshot the same current Linux Node bytes into a fresh owner-only
directory before credentials. Verify the copy, execute it to bind its
path and version, and put only that directory first on the job-local
path. Record original and copied Node modes and hashes plus the actual
sidecar mode and hash. Keep shared cache permissions and Runner security
checks intact.
- Add focused context, identity, concurrency, permission and
secret-exclusion controls. Supply the production physical cap in the
existing release-workflow fixture.
- Fix the existing Kimi cancellation fixture to await the complete
readiness sentinel within the same 100 by 10 ms bound. Keep its
cancellation and cleanup assertions.

## Verification

- The original empty-ID failure reproduces locally on the actual helper
with all 41 standing cases and zero job-run API reads.
- All 63 Node publication, projection, metrics, permission and
workflow-security tests pass. The repaired manual context flows through
the actual selection helper to case 194. Wrong launch, duplicate IDs,
malformed values and command failures still stop selection.
- The original catalog prefix rejects configured 100 / hosted 57 /
requested 2. The repaired prefix admits 2, clamps a missing request to
57, keeps legacy 100, and rejects requests above either ceiling.
- The full release-policy CI command passes locally: 72 release-workflow
tests and 107 companion tests. The first sandbox run could not bind its
loopback fixtures; the same tests passed with local-network access. The
prior-head CI fixture omission is retained and fixed by supplying the
same physical cap as production.
- Both full static Node file groups pass locally: 418 passed and 11
existing opt-in skips, at concurrency 2. Initial runs used stale
dependency links and failed. Correcting only isolated ignored links to
declared cached dependencies and pnpm 9.15.4 removed those setup
failures. Local Node 26.4 differs from CI Node 24; the exact-head CI
below provides canonical verification.
- `actionlint .github/workflows/runner-protocol-live-evals.yml` and `git
diff --check` pass.
- The new staging helper passes all four tests as a nonroot Linux user
with networking disabled. An actual retained Runner comparison
reproduces the qualified-artifact rejection with Node mode 0777. The
same Node bytes at 0500 in a 0700 directory pass that launch boundary
and reach a closed provider session. The local turn still fails without
credentials or network; this is permission proof, not a protocol case
pass. Local Linux Node 24.21 differs from the hosted 24.22.1 executable.
- Exact final head `8e06ee5611` [repository
CI](https://github.com/paperclipai/paperclip/actions/runs/41892896405)
completed successfully. Typecheck, build and all four Runner lanes
passed. Runner package tests passed 4950 tests in 297 files, with 10
existing skips. Fresh exact-head Greptile review is 5/5 with no
actionable findings or open threads. Prior `59378f` CI Kimi
empty-sentinel race and canceled chat worker remain retained.
- The patched manual workflow has not run against live TestOps. A
separate service-triggered probe supplies its own ID. The service probe
used the original concurrency guard with a temporary configured ceiling
of 57. After the probe, the original repository ceiling 100 was restored
with no active campaigns. That probe does not verify this patched
concurrency code. The next launch waits for the protected merge and
source/lock verification.

## Risks

An incompatible official CLI output fails closed. The helper never
evaluates the output or records its unknown variables. The existing job,
workflow URL, source, plan and provider checks remain required. An
omitted concurrency request now uses the lower of the configured and
physical ceilings. Explicit requests above either ceiling still fail.
The worker now executes a private copy of its existing Node binary. A
copy, mode, hash, path, version or sidecar mismatch stops before
credential access. The copy does not make dynamically linked Homebrew
Node portable; this workflow path is Linux only. This change does not
alter the provider roster, policy, limits, retries or result grading.

## Model Used

OpenAI Codex, GPT-6. The exact hosted model ID and context window are
not exposed in this session. Used reasoning, code editing, repository
inspection and tool execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-11-04 15:22:59 -06:00
DottaandPaperclip 0c2a776717 fix(evals): distinguish message objects from destinations
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-04 15:20:26 -06:00
DottaandPaperclip 99e56b32d7 fix(evals): preserve omitted credential objects at chat requests
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-04 15:18:58 -06:00
Nicky LeachandClaude Opus 5.5 be20d4ad97 fix(projects): keep projects on the storage chosen at creation (#19036)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A project stores its work in Core/local storage or in a storage
plugin
> - Today, Core finds the installed default each time it reads a project
> - So installing a default storage plugin moves existing projects,
without a saved change
> - Agents that cannot reach the new storage then lose access to new
workspaces
> - This pull request makes Core apply the default once, when it creates
a project
> - The benefit is that a project stays on its storage until a person
moves it

## Linked Issues or Issue Description

No issue exists. This describes the problem. I searched open pull
requests for related work and found no duplicate.

**What existing behavior does this improve?**

How Core chooses the storage for a project when a plugin claims
`defaultForProjects`.

**Current behavior**

Installing such a plugin changes the storage of every existing project
with no explicit choice. Disabling the plugin changes it back. Nothing
is saved, so the project has no fixed owner. Execution environments and
storage are separate contracts, so an agent can lose access to new
workspaces on a project that it used before.

**Proposed behavior**

Core saves the storage choice on the project when it creates the
project. A project with no saved choice uses Core/local storage. A
plugin-backed project stays attached to its plugin. A Core/local project
can move to a plugin.

**Reason and benefit**

A project stays on its storage until a person moves it, so installing or
disabling a plugin does not change where work goes.

**Alternatives considered**

- Keep the default at read time and add a process that saves a choice on
existing projects when a plugin arrives. This adds a background step and
a race with new projects.
- Add a new database column. The existing policy field already holds the
choice, so a migration is not necessary.

## What Changed

- `resolveProjectWorkspaceProvider` reads only the saved choice. It no
longer applies an installed default.
- `withDefaultProjectStorage` applies the default when Core creates a
project, including a personal private project. With more than one
default, a project with no explicit choice is refused. This is the
existing conflict rule, now applied at creation.
- `projectService.update` refuses a change of storage on a plugin-backed
project. An update that omits the field keeps the attachment.
- Plugin lifecycle delivery sends project events to a storage provider
only for projects attached to it, and for deleted projects. Core
acknowledges the other events for that provider. Other plugins continue
to receive all project events.
- The project settings control has no "Inherit" option. It is locked on
a plugin-backed project.
- `ProjectWorkspaceProviderConfiguration.source` no longer has the
`"default"` value.
- A project with no repository or folder ignores its saved storage
choice when a task runs. A storage plugin cannot manage this type of
project, so its tasks use a plain task directory. Without this change,
the second run of a task reset the agent session, and remote agents
could not use an empty task directory.
- The plugin SDK README and the plugin authoring guide describe the new
rules.

## Verification

- `npx vitest run
server/src/__tests__/project-workspace-provider.integration.test.ts
server/src/__tests__/plugin-lifecycle-delivery.test.ts
server/src/__tests__/plugin-workspaces.integration.test.ts
packages/plugins/sdk/tests/workspace-provider.test.ts`: 93 tests pass.
- `npx vitest run src/components/ProjectProperties.concurrency.test.tsx`
in `ui/`: 9 tests pass.
- `pnpm --filter @paperclipai/server typecheck` and `pnpm --filter
@paperclipai/ui typecheck` pass.
- `npx vitest run` on `heartbeat-session-preparation`,
`workspace-preparation`, `remote-workspace-source`,
`project-workspace-provider.integration`, and
`plugin-workspaces.integration` in `server/`: 231 tests pass.
- I did not run the full test suite locally. CI runs it.
- I did not test this change on a running instance.

## Risks

- **Behavior change at upgrade:** an existing project that uses a plugin
only through the installed default, with no saved choice, moves to
Core/local storage after upgrade. Its plugin files stay in place, but
new work does not use them until an operator sets
`storageProviderPluginId` on the project. This pull request has no
automatic backfill.
- **Missed lifecycle events:** a provider does not receive events for a
Core/local project. If the project moves to that provider later, the
update event starts its reconcile.
- **One-way move:** a person cannot move a plugin-backed project back
through the API or the UI. This is intended. If a plugin is disabled,
the project fails closed, as it does today for an explicit choice.
- This pull request does not stop a person from assigning an agent to a
project with storage that the agent cannot reach. That is follow-up
work.
- This changes core plugin behavior, and I did not discuss it in `#dev`
first. Maintainers can tell me if they want a discussion before review.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), through Claude Code, with tool
use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-04 13:07:29 -08:00
DottaandPaperclip 8a0e7ab511 fix(runner): preserve ordinary append for unscoped governed terminals (#19044)
## Thinking Path

> - Paperclip manages work done by AI agents.
> - A native turn can park on a saved human question or approval.
> - Its terminal event can return a committed receipt before observers
finish.
> - That option requires an explicit turn ID at the event port.
> - The protocol also permits events without a turn ID.
> - The runner requested the option for those events, which the port
rejected.
> - This change keeps those events on the ordinary append path.

## Linked Issues or Issue Description

Refs: #19025.

**What happened?**

The native governed-wait observation can accept a schema-valid
`item.completed` without a turn ID. If the later terminal also has no
turn ID, the runner compares two absent IDs and requests
`deferPostCommitObservers`. The event port rejects that option. The run
then fails with `governed_wait_provider_terminal_unproven`.

**Expected behavior**

Only a terminal with an explicit turn ID can request the early receipt.
An unscoped terminal must keep the ordinary append behavior and await
its observers. The event port must retain its turn identity check.

**Steps to reproduce**

1. Create a pending human-only interaction from the run.
2. Pass a schema-valid completed tool event without a turn ID through
the real governed-wait observation.
3. Supply an interrupted terminal without a turn ID during the
cancellation drain.
4. Run the actual native runtime with the PostgreSQL event port. The
parent code rejects the deferral request and reports a failed terminal.

**Paperclip version or commit**

Reproduced on `4ea53f9eb5a24de10db3dfda18456132987c4af8`. The follow-up
was identified by a review published after #19025 merged. No observed
provider emission or original-task recovery is claimed. I checked
current master and related PRs before this correction.

**Deployment mode**

Built from source. The regression uses real PostgreSQL and a
deterministic provider boundary; it makes no model call.

## What Changed

- Require a truthy terminal turn ID before the runner requests deferred
observers.
- Keep ordinary append behavior, the port identity check and all
timeouts unchanged.
- Exercise the production governed-wait observation with a persisted
pending interaction and the real event port.
- Retain exact-turn observer and checkpoint controls. Add a direct
rejection control for unscoped deferral.
- Document that unscoped terminals await their observers.

## Verification

- First meaningful red: the actual observation, native runtime and
PostgreSQL port fail on the unmodified parent runtime. The new case
reports `NativeProviderTerminalFailure` in 143ms; this is not a setup or
deadline failure.
- Focused regression and authority controls: 12 passed, 69 filtered.
Full runtime suite: 203 passed. Full PostgreSQL port and timing suites:
84 passed.
- The tests leave the human-only interaction pending with continuation
policy `none`, accept one result, and start only one turn. Existing
scope, invalid evidence, checkpoint, abort, duplicate and restart
controls remain.
- Exact candidate: `8840ea2c9c308e5a60423e67fb3612d24029ba84`.
Repository-wide `pnpm -r typecheck`, `pnpm build` and `pnpm
check:module-boundaries` passed on this exact head. `pnpm test:run` is
still in progress and is not reported as passing. Current-head CI
passed: 55 success, 3 skipped and 1 neutral, including required `ci /
verify` and `ci / e2e`. Greptile and Ponytail both reviewed this exact
head at 5/5 with no blocking findings. The native review was
source-only. No unresolved review threads remain.
- No UI, database schema, agent permission or mapped user entry point
changes. No deployment, live retry or provider replay was performed.

## Risks

Unscoped terminals retain the existing observer wait and can still fail
if it exceeds the existing drain deadline. They do not gain an early
receipt. The change does not supply a missing turn ID or weaken source,
company, run, session, cancellation or transaction checks. A PRP
terminal is not physical-stop or accounting-completion proof.

## Model Used

OpenAI `gpt-6.1-sol`, with code and shell tools. The host does not
expose the reasoning mode or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-04 15:07:06 -06:00
DottaandPaperclip f00b2a5fa2 fix(evals): normalize solicitation contrast boundaries once
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-04 15:03:59 -06:00
DottaandPaperclip 64fca2061a fix(evals): reset credential requests at explicit contrasts
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-04 15:03:59 -06:00
DottaandPaperclip 566b73729a fix(evals): retain governing credential request clauses
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-04 14:53:32 -06:00
DottaandPaperclip fc33bb0eb8 fix(evals): share positive credential request classification
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-04 14:44:30 -06:00
DottaandPaperclip c99bcd89fe fix: preserve Product TestOps evidence and CI readiness (#19000)
## Thinking Path

> - Paperclip manages agent work through governed product workflows.
> - Product E2E tests check those workflows through the app and Runner.
> - A trusted reporter sends retained results to TestOps.
> - Free-form metadata and a short upload observer can lose safe, valid
evidence.
> - UUID navigation can also discard task metadata before the canonical
query returns.
> - Hermetic CI files start child processes that need CPU capacity
within their readiness limits.
> - This change closes the reporting contract, retains authenticated
core aliases, and reserves CI CPU capacity.
> - Original case identities, conditions, assertions, verdicts and
deadlines remain intact.

## Linked Issues or Issue Description

Refs #18983. This addresses its late [metadata disclosure
finding](https://github.com/paperclipai/paperclip/pull/18983#discussion_r4470789114)
and [upload readiness
finding](https://github.com/paperclipai/paperclip/pull/18983#discussion_r4470790366).

**What happened?**

The reporter accepted model IDs and matcher names through syntax
filters. A secret-shaped string could pass those filters. First-task
results also use their top-level model field for an observed alias.
Treating it as configured metadata can reject valid retained evidence.
The upload observer stopped after 22 seconds of waits.

[Required CI on the earlier
head](https://github.com/paperclipai/paperclip/actions/runs/41886885193)
also found two real browser title failures and five native package
readiness failures. A deterministic control reproduced metadata loss
during UUID-to-identifier navigation. A separate fixed-resource control
reproduced startup deadline failures with four concurrent hermetic
fixtures, while one and two fixtures passed.

**Expected behavior**

Reject unknown transmitted metadata before creating upload files.
Preserve every original matcher verdict and attempt receipt. Observe
server processing without rerunning the product. Keep authenticated task
metadata visible through canonical navigation. Reserve capacity for
hermetic child processes without changing the original readiness
deadlines.

**Steps to reproduce**

Use a secret-shaped model ID or cleanup state in retained evidence,
return an observed first-task model alias, or let upload processing
settle after 40 seconds of virtual time. Hold the canonical core query
pending after an authenticated UUID read. For the CPU control, run the
same providerless startup/recovery fixture alone, four times
concurrently, and twice concurrently under a fixed 4-vCPU quota and its
original 8-second readiness limit.

## What Changed

- Bind configured model receipts to each exact declared case model. Four
first-task cases use captured agent configuration; their observed
aliases and fallback text remain digest-only. A missing capture stays
null / Not recorded. The other 16 cells use their configured profile
model. Reject wrong or unknown configuration before any upload file
exists.
- Keep the existing model distribution: 18 Codex cells on `gpt-5.6-sol`,
configured-repository case 9 on its existing `gpt-6.1-sol` override, and
task-workspaces case 54 on `openrouter/deepseek/deepseek-v4-flash-0731`.
The repository override exists at the [original canary
source](https://github.com/paperclipai/paperclip/blob/86814c6550f52e99ede150a883bdc773cbb540e3/tests/runner-e2e/catalog.ts#L1148).
No model configuration changes.
- Allow only existing Matcher kinds, closed failure classes, and cleanup
states `passed`, `failed`, or `not_started`. Publish each matcher's
kind, ordinal, original boolean verdict and SHA256 path receipt. Full
conditions remain in the linked original workflow artifact; raw dynamic
path bytes do not enter TestOps.
- Observe the same upload for at most 55 seconds of poll waits. Product
scenarios, provider retries and scenario deadlines do not change.
- Retain task core data under only the authenticated response's own UUID
and identifier core keys. Preserve existing canonical data. Keep
arbitrary references and full-detail keys empty until their own reads.
- Retain a first-failure trace for the existing Cloud recovery test.
Keep its assertions, timeout, retries, mock data and artifact policy
unchanged. This adds diagnosis evidence; it does not claim to fix that
separate Cloud title timeout.
- Reserve half the available CPUs for Runner package CI workers, with
the existing upper cap of four. A 4-vCPU host uses two workers; an
8-vCPU host still uses four. Local defaults and test selection do not
change.

## Verification

- Reporting controls first reproduced the two original gaps. All 14 Node
contract tests now pass. Unknown configuration creates neither a result
directory nor an export summary. A model valid for a different selected
case is also rejected. Original functional failures and earlier
accounting receipts survive.
- The catalog compatibility test passes for all 20 selected cells, exact
model tuples, real pure grader outputs and actual
`projectFirstTaskResult` shapes. It covers observed aliases, fallback
text, absent evidence and wrong or secret-shaped configured metadata.
These are contract controls, not live journey passes.
- Virtual-clock controls accept a correct result that settles at 40
seconds and reject a persistently wrong verdict after 55 seconds. They
make no product or provider calls.
- Two new core-cache controls failed before repair. The query/cache
suites now pass all 8 tests. They hold canonical refresh pending and
verify title/company retention, newer canonical data, full-detail
enrichment and unrelated keys. UI typecheck and token gates pass.
- The unchanged exact UUID cross-company link/upload browser test passed
once after repair: 5.8-second scenario, 32.0-second complete command.
The unchanged exact Cloud task-entry diagnostic passed once: 5.4-second
scenario, 23.3-second complete command. Its original canonical-only
timeout remains unexplained. All five Cloud tests still collect after
the trace setting.
- The providerless Linux control used the retained Runner and compiled
controller from source `3a2249dd`, with a fixed 4-vCPU quota and
original 8-second readiness limit. Solo passed 1/1; four concurrent
fixtures failed 4/4; two concurrent fixtures passed 2/2. At each
failure, Rust had accepted command 2 but startup was still pending;
checkpoint and published status agreed, one connection was
authenticated, no replay backlog existed, and the sidecar had not
launched. Passing controls preserved the expected startup failure
through controller replacement. No network or credentials were
available.
- That control uses Linux amd64 emulation and Node 24.21; hosted CI used
Node 24.22.1. It supports the worker-cap change, but does not prove that
the original `m5n.xlarge` host caused every CI failure. The original red
attempts remain recorded.
- Actual config-load controls verify 1 CPU → 1 worker, 2 → 1, 4 → 2, 8 →
4 and 16 → 4, with no local, timeout or retry override. The isolated
policy change preserves the same 297 test-file inventory; integration of
current master `89703c3ff7` retains all 298 current native package
files. The original PR package/native command plans remain unchanged.
- Final head `877e28da32f2d2f0d12906c89aff085e2c6ea943` passes both
required checks, `ci / verify` and `ci / e2e`, in [CI run 41897677652,
attempt
2](https://github.com/paperclipai/paperclip/actions/runs/41897677652/attempts/2).
The native package passes 4,950 tests with 10 existing skips. All eight
browser shards pass; the Cloud shard passes all 41 tests, including all
four unchanged recovery cases.
- Attempt 1 preserved 46 successful jobs but its one-file heartbeat
recovery shard hit the original 20-minute job limit without a named
assertion or completion verdict. Its cause remains unidentified. One
explicitly bounded debug replay retained those 46 successes and ran only
that leaf and its dependent rollup. The unchanged file passed all 412
tests (346.91-second file, 371.34-second Vitest command, 7m11s job).
Source, cases and deadlines did not change. This establishes
intermittency, not a causal repair. The original timeout and debug pass
both remain recorded.
- Fresh [Greptile
5/5](https://github.com/paperclipai/paperclip/pull/19000#issuecomment-6637604918)
and [Ponytail
5/5](https://github.com/paperclipai/paperclip/pull/19000#issuecomment-6637489454)
reviews name exact head `877e28`. The single review thread is resolved.
Live TestOps bootstrap and complete 20-case acceptance remain held until
reviewed delivery. CI and focused controls do not count as product plan
passes.

## Risks

- Future model or Matcher kind changes require a reviewed
reporting-contract update.
- TestOps shows a closed Matcher kind and ordinal. The path digest binds
the original condition in retained evidence.
- Core alias data retains the existing authenticated query-client and
server company boundaries. It never becomes full-detail evidence.
- Fewer CI workers can increase one job's duration. The full suite must
verify the final policy. The paid evaluation matrix and its parallelism
do not change.
- The original Cloud title timeout and the first-attempt heartbeat CI
hang do not have proven causes. Current unchanged assertions pass, but
this does not establish repeatability. The Cloud test now retains its
first-failure trace. A successful upload or focused control alone cannot
establish a complete product pass.

## Model Used

OpenAI Codex with reasoning, tool use and code execution. The host does
not expose the exact model ID or context window to this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` OR (b) described the issue in-PR following the relevant issue
template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-04 14:41:28 -06:00
DottaandPaperclip 513c03cb3e fix(e2e): configure supported rolling agents with warm policy (#18994)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Rolling tests must exercise the lifecycle they claim to cover.
> - The warm lane selected a warm environment.
> - Its profiles also set an explicit per-turn agent policy, which took
precedence.
> - The test then waited for a live workspace that per-turn cleanup had
correctly retired.
> - This pull request sets warm agent policy for providers admitted by
the production warm capability.
> - The matrix now checks effective lifecycle and preserves per-turn
coverage for every qualified native profile.

## Linked Issues or Issue Description

Builds on merged #18982 for the source controller dependency link. This
PR now targets master.

**What happened?**

The warm rolling fixture built a Codex agent with `lifecycleMode:
per_turn` and an environment with `runnerLifecycleMode: warm`. The
explicit agent policy won. The initial task completed, its disposable
workspace retired, and the fixture waited for a live file before it
reached deployment.

**Expected behavior**

Warm rolling, legacy migration, remote-file and controller-outage cells
must create actual warm agents. Only providers supported by the
production managed warm-session capability belong in that lane. All
qualified native profiles retain per-turn coverage.

**Steps to reproduce**

Build the old warm Codex cell's agent and environment and compare their
lifecycle policies. The new effective-policy regression fails on the old
catalog and passes on this branch. No duplicate exact-scope warm-fixture
PR was found.

## What Changed

- Share the production managed warm capability filter and explicit warm
agent policy across rolling, legacy migration, remote-file and
controller-outage suites.
- Check effective agent and environment policies and update matrix
coverage/counts.
- Read completed warm turns from the exact saved snapshot while
independently requiring the retained, bound workspace location.
- Keep both rolling lanes separate from the outage simulation and
exercise their assembled supervisor requests.
- Version the corrected definitions and keep every qualified native
profile in the per-turn lanes.

## Verification

- Baseline regression: failed against the unchanged catalog; retained
before the fix.
- Fixed catalog/controller selection: 39 checks passed.
- The first real warm-policy attempt completed and saved its workspace
but timed out because the file oracle demanded live content; retained
trace responses prove the API returned the correct saved snapshot with a
retained lease.
- A new regression fails on the previous reader and passes after
separating saved file source from retained versus released location.
Focused catalog, reader and rolling checks: 81 passed.
- The next warm attempt reached replacement and exposed a second fixture
defect: warm also selected a SIGKILL outage, which the supervisor
rejected alongside rolling mode. The failure remains retained; its four
owned sandboxes, temporary key and local bucket were deleted.
- An outage-selection regression failed before the fix; 57
outage/supervisor checks pass afterward, including assembled requests
for both rolling lanes. The fresh warm Codex run passed all 56 checks in
8.5 minutes: original run/session/runner/lease retained, authenticated
takeover, both predecessor homes removed, later native continuation and
saved file readback. Two replacement receipts recorded 742 ordinary
reads and 75 writes, all 75 read back, with zero HTTP probe failures. A
transient `native_workspace_sync_out_failed` phase recovered before
successful finalization and remains retained. This source-lane result
does not claim released-image or hosted Railway qualification.
- The invalid earlier attempt was interrupted before deployment. Its
four archived disposable sandboxes were deleted and verified absent; its
temporary key and local bucket were removed.
- Broad local typecheck reached stale shared database declarations in
this checkout's linked build dependencies. The fixture's own type error
was corrected; clean-head CI remains required. No full build is claimed
locally.
- An incidental report-catalog test selected by a substring filter
failed to generate its normalized output. That failure is retained
separately; the focused catalog/controller checks pass.

- Review follow-up: fixed the same policy conflict in the base
file-storage, legacy migration and controller-outage suites. The roster
now has 822 valid cells; unsupported warm combinations remain covered
per-turn. Updated the explicit supervisor-request control to use a
supported warm provider. Focused catalog, file-reader, migration,
rolling and supervisor checks pass (133 checks).
- Earlier full CI passed at dd5aa71 using the built-in GitHub-hosted
fallback after self-hosted runner shutdowns. That result does not
qualify this new head; fresh CI and review are required. The final head
now has a 5/5 review and passing full CI (run 41901807639, attempt 2).
Attempt 1 hit two unrelated port collisions: the unchanged managed Codex
credential lease test and CLI company-server startup on a busy requested
port. Both passed on one failed-job retry. Those failures remain
retained.

## Risks

This changes test setup and coverage. It does not change agent runtime
defaults or provider capabilities. Warm cells use the production
capability boundary, while every qualified native profile keeps its
per-turn cell.

## Model Used

OpenAI Codex, exact model ID `gpt-6.1-sol`, with reasoning, tool use and
code execution. The host does not expose context window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either linked existing issues or described the issue in-PR
following the relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links
- [x] My branch name describes the change and contains no internal
ticket id or instance-derived details
- [x] I have run the relevant tests locally; failures and limits are
retained above
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-04 14:40:45 -06:00
DottaandPaperclip 5aa9e33492 fix(evals): preserve imperative boundaries in credential warnings
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-04 14:30:34 -06:00
4ea53f9eb5 fix(server): resolve issue identifiers in file-resources routes (#16352)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents and the board read task workspace files through the
`/api/issues/:issueId/file-resources/*` routes
> - Clients can identify a task by its UUID or by its human identifier,
for example `ABC-230`
> - The issues router changes identifiers to UUIDs, but the
file-resources router does not
> - Thus an identifier goes into a `uuid` comparison, Postgres rejects
the query, and the route returns 500
> - This pull request resolves identifiers in the file-resources router
and returns 404 for malformed ids
> - The benefit is that identifier links work, and bad ids give a
correct 404 and no server error

## Linked Issues or Issue Description

**What happened?**
A request to `GET /api/issues/ABC-230/file-resources/list` returned 500.
The server log showed `invalid input syntax for type uuid` from
`workspaceFileResourceService.getIssue`. Sentry recorded this error on
hosted staging.

**Steps to reproduce**
1. Create a task that has an identifier, for example `ABC-1`.
2. Send `GET /api/issues/ABC-1/file-resources/list?workspace=project`.
3. The server returns 500. The same request with the task UUID returns
200.

**Expected behavior**
The route accepts the identifier, as the other `/api/issues/:id` routes
do. An unknown or malformed id returns 404.

## What Changed

- Added `resolveIssueId` to `WorkspaceFileResourceService`. It changes a
task identifier to the task UUID. It does not check visibility.
`getIssue` still does the privacy-scoped read.
- Added a `router.param("issueId")` handler to the file-resources
router. It calls `resolveIssueId`, the same as the issues router does
for its routes.
- `getIssue` now returns 404 before the query when the id is not a UUID.
This stops the Postgres cast error for all callers.
- Added a regression test for identifier, lowercase identifier, private
task, unknown identifier, and malformed id on the `content`, `list`, and
`availability` routes.
- Updated the test service fakes with the new method.

## Verification

- `pnpm vitest run src/__tests__/file-resources.test.ts
src/__tests__/file-resources-git-scan-load.test.ts` in `server/`: 55
passed.
- Without the source change, the new test fails with `expected 200 "OK",
got 500 "Internal Server Error"`. This reproduces the production error.
- I did not run a full local `tsc` because the sandbox did not have
sufficient memory. CI runs the typecheck.

## Risks

- Low risk. The change adds one indexed lookup on `issues.identifier`
(unique index) only when the path value has the identifier format.
- The identifier lookup does not check visibility. `getIssue` then
applies the privacy-scoped read, so a private or unknown task gives the
same 404 as before. The route does not show if a task exists.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edit, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Paperclip <noreply@paperclip.ing>
canary/v2026.1104.0-canary.13
2026-11-04 12:30:26 -08:00
3cdb367369 fix(acpx): report how the sandbox agent process session ended (#16366)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Hosted `claude_local` runs use the ACPX engine. The agent process
runs in a sandbox, and a host-side process session bridge carries the
ACP stream.
> - Some hosted turns fail with only "The connection to Claude was
lost." (`acpx_turn_failed`, terminal session category `connection`). The
failure report shows one pending `execute` tool. It does not show why
the agent process went away.
> - The bridge already records how the session ended: the remote exit
code and signal, or a host proxy close. But it writes that evidence only
to the run log. The run failure report never receives it.
> - Without it, we cannot tell an OOM kill (`SIGKILL`) or a provider
process exit from a bridge or proxy close. Each cause needs a different
fix, so we must not guess.
> - This pull request keeps the first terminal evidence on the bridge
handle and copies it, as closed values, into the run failure report's
execution context.
> - The benefit is that the next recurrence names the cause, and the
correct fix can follow.

## Linked Issues or Issue Description

**What happened?**

Hosted `claude_local` runs on staging and production failed with "The
connection to Claude was lost." One run failed about 4 minutes in, 7–10
seconds after its last ACP event. Another failed after about 3.5 hours
with no ACP events for most of that time. Both had one pending `execute`
tool. The report had no information about how the sandbox agent process
ended.

**Expected behavior**

The run failure report says how the sandbox agent process session ended.
Examples: the remote process exited with a code, the remote process was
killed by a signal, or the host proxy closed.

**Steps to reproduce**

1. Start a hosted ACP `claude_local` run on a sandbox target.
2. Kill the agent process in the sandbox while a tool runs, for example
with `kill -9`.
3. See that the run log has an `ACP process session terminal:` line, but
the run failure report has no matching field.

**Paperclip version or commit**

`master` at 1b58ac6.

## What Changed

- `startAdapterExecutionTargetProcessSessionBridge` keeps the first
terminal evidence it records (`source`, `outcome`, `exitCode`,
`signal`). The new optional handle method `readTerminalEvidence()`
returns it. The run log line does not change.
- The closed source and signal labels move to
`acpx-engine/constants.ts`. The bridge and the server reader use the
same list.
- The ACPX engine adds `acpProcessTerminalSource`,
`acpProcessTerminalOutcome`, `acpProcessTerminalExitCode` and
`acpProcessTerminalSignal` to the turn activity diagnostics in
`resultJson`. It leaves out an unknown exit code or signal.
- `collectRunFailureDiagnostics` copies these fields into the execution
context. It accepts only closed labels and an integer exit code from 0
to 255. If the source is not a closed label, it drops all the fields.
- `doc/observability.md` documents the new fields next to the other ACP
activity fields.
- Tests: bridge evidence for a remote exit, a missing command, and an
authenticated proxy close; the engine summary helper; and the server
reader with valid and invalid values.

## Verification

- `vitest run src/acpx-engine/ src/execution-target-sandbox.test.ts` in
`packages/adapter-utils`: 22 files, 579 tests pass.
- `vitest run src/services/__tests__/run-failure-diagnostics.test.ts` in
`server`: 121 tests pass.
- `tsc --noEmit -p packages/adapter-utils`: passes.
- I type-checked the changed server files with a narrow config. The only
errors came from workspace packages that were not built locally
(`@paperclipai/plugin-sdk`, `@paperclipai/paperclip-runner`). A full
server `tsc` ran out of memory in my sandbox. CI runs the full check.

## Risks

- This change adds diagnostics only. It does not change retry, recovery,
or classification behavior.
- Every new field is a closed label or a bounded integer. No message,
command, path, payload, or provider text goes into the report.
- The host records a `proxy_close` from a socket close event. If the
turn finalizes before that event, the fields are absent for that run.
When the fields are absent, the report has the same data as it has
today.
- `readTerminalEvidence` is optional on the handle type, so other handle
implementations and test doubles still compile.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with tool use
and code execution. It read the repository, traced the failure path,
edited the code, and ran the tests.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Related: #16360 bounds hung MCP tool calls (ACP kind `other`). This
change is separate. It covers the sandbox agent process ending while a
tool runs.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: devinfoley <devinfoley@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com>
Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-04 12:15:43 -08:00
DottaandPaperclip 07eb8e67ce fix(runner): deliver configured task environment to OpenCode
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-04 14:14:17 -06:00
Dotta 65978bb6d0 fix(evals): retain qualified nonhuman handoff actors 2026-11-04 14:14:02 -06:00
DottaandPaperclip 68fa733d47 fix(evals): bind personal publishing proof to native output receipt
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-04 14:13:51 -06:00
Dotta 6d6dc6e397 fix(evals): retain correlative success and known-source takeover 2026-11-04 14:13:29 -06:00
Dotta 1ad2507b85 fix(evals): distinguish timeline actors from deadline adjuncts 2026-11-04 14:13:29 -06:00
Dotta 9f8394c532 fix(evals): bound timeline object and result assertions 2026-11-04 14:13:29 -06:00
53735db7de fix(retry): resume verified preparation failures (#19004)
## Thinking Path

> - Paperclip must preserve saved agent work while proving earlier
execution stopped.
> - A deliberate task Retry can observe a newer failure while an older
preparation still holds execution.
> - That older preparation can also have a finalized empty object log
despite never reaching the provider.
> - Binding only the latest run and requiring every log field to be
absent keeps valid recovery deferred.
> - This change follows the exact persisted predecessor chain and binds
current Retry authority to eligible holds.
> - It recognizes only an exact finalized empty-log receipt within the
existing strict preparation proof.
> - One fresh user turn can then start with saved work and the existing
ownership, stop and accounting checks.

## Linked Issues or Issue Description

Refs #18989, #18860 and #18997. The first waits for predecessor cleanup
during automatic dispatch. The second selects the committed native
session owner. The third retains admitted Retry authority when no source
recovery action exists. None passes current board Retry intent to a held
failed predecessor. A search of open repairs found no duplicate.
ROADMAP.md was checked; this is a recovery bug fix.

**What happened?**

A failed native preparation retains a legacy run row and an inherited
session. Its remote workspace is later saved and verified. An automatic
successor also fails during preparation. Board Retry observes that newer
run, but the effective hold belongs to its predecessor. The Retry does
not bind the predecessor's recovery generation. The native
historical-source guard then keeps the request deferred.

**Expected behavior**

Carry the deliberate Retry through its verified same-company, same-task
and same-agent persisted predecessor chain. Start one fresh user turn
after every source passes current execution, stop and saved-work checks.
Keep incomplete or unrelated proof held. Preserve prior work and
recorded action outcomes.

**Steps to reproduce**

1. Fail native preparation with an inherited session and a remote
workspace.
2. Finish its real workspace recovery and stop receipts.
3. Save a newer failed automatic successor with `retryOfRunId` bound to
that source.
4. Submit ordinary task Retry for the newer run.
5. Before the change, no successor is admitted. The request remains
deferred behind the older hold.

**Paperclip version or commit**

The regression fails on master
`039e0dad2c51c80251e1302a19c5566e039c357a`, which includes #18989.

**Deployment mode**

Source checkout on macOS arm64 with real PostgreSQL, HTTP task Retry and
the assembled heartbeat service. The remote filesystem boundary uses
temporary files. Hosted rollout and an original-task Retry are not yet
verified.

This PR incorporates the reviewed empty-log follow-up #19021, preserving
its commits and regression controls. Both parts are required for hosted
preparation recovery, so the combined revision is qualified and deployed
together.

## What Changed

- Accept a finalized zero-byte `s3_segments` log only when its reference
binds the exact company, agent and run, its digest matches empty
content, compression is false, and no output progress or nonempty
excerpt exists. Add a real-recorder HTTP regression and 14 negative
controls. Unknown or nonempty logs stay held.

- Read a complete persisted Retry chain in one recursive query, limited
to 32 runs. Reject missing, cyclic, newer, unfinished, cross-company,
cross-task and cross-agent sources. Transcript hints grant no authority.
- Bind eligible historical recovery records to the current Retry intent.
Keep complete or running recovery budgets. Leave successful ancestors
and mismatched recovery-source records unchanged.
- Recheck that same chain when selecting an ancestor's fresh-user
strategy.
- Admit only a held ancestor through the shared native preparation
proof. Keep the closed preparation failure codes, no-provider state,
profile, input, result, output and accounting checks. Treat inherited
session input as prior context and accept the exact bounded host start
marker. Verify current ownership, saved work and stop receipts before
forcing a fresh session.
- Add HTTP and PostgreSQL regressions and document the rule.
- Give the HTTP Retry nudge and explicit test worker the same real
heartbeat facade and provider boundary. Establish the revoked catalog
receipt before either can dispatch.

## Verification

- Combined integration head `91f021f233151192f69dd7a035d083abcc67c47c`
requires fresh complete CI and review; qualification is in progress.

- Repair validation: all **six** recovered preparation provider-boundary
cases pass. The catalog cases still require the exact access refusal,
zero provider inputs and zero provider turns. The success case still
checks preserved file bytes and one native turn.
- [Previous exact-head
CI](https://github.com/paperclipai/paperclip/actions/runs/41883421120/job/126785508136)
failed the catalog case with `remote_workspace_transport_unsupported`.
The route created a default heartbeat that could race the injected
worker and allocate outside the controlled transport. The repaired test
shares its real heartbeat across both dispatchers and holds the agent
start lock until the revoked catalog receipt is committed. It tracks and
drains both Retry nudges during teardown. It changes no production
access check or expected refusal.
- The first broad local reproduction hit a full local disk. It passed
201 cases and failed 86, including explicit manifest free-space errors.
Generated build caches were removed from these isolated checkouts. The
owning run passed 670/671 cases. Its one failure was the saved-Files
loader refusing insufficient temporary disk space. After cleanup, that
unchanged test passes in isolation. The final shared-facade and
start-lock version passes all six dispatch cases. Both legacy
continuation suites pass all 29 cases. The final fixture teardown also
retires deferred input before removing recovery proof, so global queue
sweeps cannot revisit earlier cases. All seven dispatch and saved-Files
controls pass together after that change. No disk reserve, assertion or
timeout changed. The initial local typecheck also stopped on `ENOSPC`
while materializing provider assets. Server typecheck now passes after
cleanup. No test timeout or reserve was lowered.
- Current-head CI is complete: 52 successful checks, three expected
skips, and a successful security status. Both required gates (`ci / e2e`
and `ci / verify`) passed on `0adc2e38103533dd45bcc56f957ac019642c57b1`.
[Full CI
evidence](https://github.com/paperclipai/paperclip/actions/runs/41898305521).
Greptile scored final head `0adc2e38103533dd45bcc56f957ac019642c57b1`
5/5 with no blocking findings or unresolved threads. The earlier
verification below describes the previous head.

- RED on unchanged master `039e0dad2c`: the certified-preparation
sibling passes, while the inherited-session case admits zero successors
instead of one. A second isolated RED on latest base `8f06bd4e17`,
including #18997, uses the exact final three-node test fixture. The HTTP
route returns 202, then the test observes zero successors instead of
one. All baseline production files remain unchanged. The initial
dependency/setup failures are separate from these behavioral failures.
- The positive cases cover the failed-failed-successful chain. They keep
the completed result unchanged, verify the original saved file and
validate fresh-session continuation authority. Repeated HTTP input and
concurrent resume calls keep one successor.
- PostgreSQL controls accept a complete 32-run chain. A 33-run chain and
a newer-created predecessor return no usable ancestry and do not bind
the older recovery record.
- Safety controls cover company, task and agent scope, conflicting
canonical bindings, missing and cyclic edges, transcript hints, live
execution and controllers, unknown stop, failed saving, unsafe writer
state, Stop, actor and generation changes, independent holds and
mismatched recovery records.
- Ancestor controls retain provider-event, native-input,
session-checkpoint, controller-recovery, result, process, runtime,
accounting and unknown-result guards. Host start and warning prefixes
use exact origins, sequence and payloads. A fifth extra event is
rejected.
- The first complete owning run found 22 introduced admission failures.
The initial historical-native allowance also applied to exact-source
Retry. It admitted contradictory preparation evidence. The fix now
requires a canonical ancestor and the shared strict startup proof. The
original direct-source assertions remain unchanged.
- The next focused run passed 82 cases, including both catalogue-warning
controls. One new result fixture lacked the database's required owner
tuple. Two existing controls reached their original 30-second limit. The
fixture now uses the real native task and completion-contract owner
tuple. A controlled nine-case run passes the result case, both unchanged
timed-out controls, three failure-code controls and all three positives.
The earlier timeout causes remain undetermined. All failed-run logs are
retained.
- Complete owning suites passed on previous head
`137a8eb3febf41052e112fc3c4ffe1abb92bcab7`: all 675 tests across
workspace recovery, task Retry, explicit continuation and legacy
continuation. The command exits successfully with normal teardown. This
includes every earlier introduced admission failure and the two
unchanged timeout controls. Its base includes the merged #18997
authority repair and #18982 rolling-copy packaging repair. The rebase
retained both test imports at their one adjacent-import conflict.
- Node 24.19.0 and pnpm 9.15.4. Module boundaries and `git diff --check`
pass. The initial PostgreSQL start lacked package library links after
install with scripts disabled. The documented hydration restored the
isolated dependency; that setup failure was not a behavioral red.
- Previous-head [full
typecheck](https://github.com/paperclipai/paperclip/actions/runs/41883421120/job/126785506215)
and [full
build](https://github.com/paperclipai/paperclip/actions/runs/41883421120/job/126785506094)
passed. Earlier attempts received synchronized runner shutdown signals
and a lost-runner annotation. The shutdown initiator remains unknown. A
CI owner used the documented hosted-runner failover for attempt three
and restored the original routing value after capture. That attempt
exposed the catalog failure above. Source workflows and check
requirements remain unchanged. The current head now has green CI,
Greptile 5/5, no unresolved review threads, and no merge conflicts.
Hosted rollout and the original task are not yet verified.


- Combined recovery proof: 24 of 24 cases passed on this exact head in
43.13 seconds, with normal teardown and unchanged assertions and limits.
They cover the real recorder, all 14 empty-output rejection controls,
ancestor and direct Retry, saved messages, the configured native
provider, and verified stop. The targeted TypeScript graph also passed;
the preceding integration step passed 95 safety/provider/message cases
and all 49 task Retry cases.
- The incorporated #19021 was independently red/green with the real
recorder: unchanged predecessor code returned HTTP 202 but admitted zero
successors. Its 18 focused cases, all 690 owning recovery cases, full
typecheck/build, 1,904 local chat cases, complete CI and Greptile 5/5
passed before integration. The unchanged CLI startup lane passed one
justified retry; no hosted acceptance is inferred from those results.
- Additional local qualification of the unchanged workspace fixture
alongside the native bootstrap fixture passed 288 of 289 cases. One
manual-admission HTTP case ended with `socket hang up` after 451 ms,
without an assertion or HTTP status. That unchanged case passed once in
isolation at the original 40-second limit, with normal teardown, and
passed in exact `0adc` CI in 17.2 seconds. The earlier disconnect cause
remains unknown; no assertion or timeout changed.
- An earlier CI attempt failed the newly merged native ingress fixture
because its shared Node command failed strict permission qualification.
The current base includes #19020's owned executable fixture repair; no
production permission check was relaxed.
- Integration with current master had one documentation conflict. An
automatic test merge also duplicated an app declaration; that syntax
failure was retained and fixed by reusing the configured real HTTP app.
Both new message-state assertions remain intact. Workflows and lockfile
retain master's contents.

## Risks

- A damaged or overlong chain remains held. The helper does not truncate
it and grant partial authority.
- The new query runs only for a deliberate Retry or its ancestor
strategy check. It adds no background polling, table, credential or
provider policy.
- A fresh intent is not stop or file-recovery proof. The existing owners
retain those checks. Prior effects remain fenced, and no old result is
replayed.
- Local tests do not prove hosted rollout. Live acceptance must use the
served fixed revision and the task's original saved work.

## Model Used

OpenAI GPT-6 through Codex for the original change. OpenAI GPT-6.1 Sol
(`gpt-6.1-sol`), xhigh reasoning, for the merge and CI repair. Both used
repository tools and PostgreSQL verification. Context-window limits were
not exposed by the harness.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Devin Foley <devin@paperclip.ing>
2026-11-04 14:12:58 -06:00
DottaandPaperclip 22735cfcea fix(native): preserve governed terminal receipts during observer settlement (#19025)
## Thinking Path

> - Paperclip manages work done by AI agents.
> - A native turn can yield to a saved human question or approval.
> - The provider must stop, and its terminal event must be saved.
> - The old drain waits for both the event transaction and its later
observers.
> - A slow observer can make a saved terminal look unproved.
> - This change separates the committed receipt from owned observer
work.
> - Checkpoint and result acceptance still wait for that work to settle.

## Linked Issues or Issue Description

**What happened?**

A native governed wait can fail with
`governed_wait_provider_terminal_unproven` after its exact interrupted
terminal is already in PostgreSQL. The event port waits for postcommit
accounting, output and projection callbacks before it returns the
receipt. Those callbacks can cross the provider drain deadline.

**What did you expect to happen?**

A committed terminal proves the event transaction. Its observers remain
separate work. A checkpoint or result must not be accepted until those
observers settle. A pending human-only question must keep its audience
and continuation policy.

**Steps to reproduce**

Run the real PostgreSQL control-plane regression. Commit a same-turn
interrupted terminal. Hold its postcommit observer beyond the existing
provider drain. The parent code reports an unproved provider terminal.
The fixed code retains the committed receipt and joins the observer
before checkpoint or result acceptance.

No overlapping repair was found in the current master or open-PR search.
Related [#18977](https://github.com/paperclipai/paperclip/pull/18977)
covers final conversation-response selection. It does not change this
receipt boundary. The source reproduction is consistent with the
observed failure. The exact hosted provider code is unavailable, so it
does not prove that every observed failure used this branch.

## What Changed

- Add an explicit terminal-only option to the internal event port.
Ordinary appends keep their existing behavior.
- Return that receipt only after the normal transaction and company,
agent, run, session, source and revocation checks.
- Retain the full observer promise. Join it before checkpoint and result
writes. Keep its original rejection and observe it after an aborted
join.
- Reuse the existing provider drain and checkpoint deadlines. Preserve
duplicate and restart replay without another model turn.
- Document the receipt and duplicate-observer replay boundaries. Keep
human-only and no-continuation cards unchanged.
- Assert the actual Codex facade requests deferral only for its
interrupted terminal. Isolate empty skill inventory in monitor-only
process fixtures while retaining their real scheduler, lifecycle,
selection snapshot and three-second deadline.

## Verification

- Meaningful parent red: actual native runtime and real PostgreSQL event
port. The terminal row commits, but a held observer causes
`NativeProviderTerminalFailure` with
`governed_wait_provider_terminal_unproven`.
- Current candidate `0db24ae74ee42fdf92035d8407fc095e9f73cc6a`: full
runtime suite, 203 passed. Real PostgreSQL control-plane, timing and
monitor suites, 103 passed. Runner TypeScript and full layered server
typechecks passed on the unchanged candidate source tree. The controls
cover uncommitted and foreign evidence, explicit false and failed
terminals, held and rejecting observers, abort and controller
revocation, duplicate receipts, and new-port recovery. The real pending
human-only card stays unanswered with continuation policy `none`.
Recovery does not call `startTurn` again.
- `pnpm -r typecheck`, the production server build (including the real
Runner release binary), module boundaries and vendor dependency
verification passed on `6d75`; the follow-up changes only tests and
documentation. Diff checks pass on the current candidate. The prior
`399ad3dc2b` repository-wide `test:run` was stopped as obsolete during
the general-server phase after preserving its progress (exit 130,
incomplete); its repository build was not reached. Its six completed
chat shards passed 1,904 cases, but this is prior-head evidence and not
a full-suite pass. Full current-head CI is required before merge.
- CI attempt 1 on `6d75` failed one real three-second monitor-settlement
assertion; the aggregate verify gate consequently failed. Other jobs
passed. The actual CI merge tree was
`3037bf6db162da3b0a9172046360401f8449ebc5` (`6d75` into `89703c3`). An
unchanged full-file replay reproduced three timeouts. Original-deadline
inspection found the legacy process still preparing; local phase timing
measured cold skill inventory at 6.629 seconds. The original CI did not
retain that phase detail, so its exact subcause remains unproved. The
fixture correction supplies empty skill entries for these skill-free
process tests and still invokes the selection-snapshot callback. It
preserves all assertions and the three-second deadline, with no new
drain wait. All 21 cases pass; deliberately restoring the old
blocked-status rejection still fails the real blocked-check case with
409. No CI retry was issued.
- The prior current-head native review was 4/5 with no production
correctness finding. Its requested direct facade-options assertion and
duplicate-observer wording correction are included. Current-head CI is
green (55 success, 3 skipped), including required verify and E2E;
current-head Greptile is 5/5 with no actionable findings. The old native
advisory remains tied to `6d75`, and its requested findings are
addressed. No new native assessment is claimed or required by
protection.
- The first draft broke three existing transaction-start timing
controls. The no-debt path now keeps the original synchronous start, and
all three pass. Two draft type errors were corrected. A further real-PG
red found an extra default-append assertion after commit. Removing it
restores the original committed receipt after a late abort; the same
aborted signal still cannot write a checkpoint or result. Original setup
and assertion failures remain in private verification receipts.
- Normal merge of current master
`3a2249dd5768313f04b2aa3860dfd363492eda16` has no conflicts. The
affected production source blobs did not change in that baseline merge.
- Local proof uses Node 24.22.2 on Darwin arm64, a real PostgreSQL
instance and deterministic provider-boundary fixtures. It makes no model
call. No deployed replay or original-task recovery is claimed.

## Risks

Deferred observers must remain owned. A missed join could accept
incomplete accounting or output. A rejected or stalled observer must
prevent acceptance, and an abort must prevent late checkpoint/result
writes. The tests exercise these boundaries. The port keeps the existing
best-effort trace-publication handling. This change does not turn a PRP
terminal into physical-stop or accounting-completion proof.

## Model Used

OpenAI `gpt-6.1-sol`, with code and shell tools. The host does not
expose the reasoning mode or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-04 14:11:03 -06:00
c3715e74a6 perf(ci): let the public-install registry reuse warmed scoped metadata (#16838)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Every pull request runs the `ci / Build` check. This check was the
slowest check in recent green runs: 432s in run 39254079691, 480s in run
39251304886 and 472s in run 39245233967.
> - The slowest step in Build is "Verify built-in Grok from a clean
public npm install" (162s and 207s). Its consumer `npm install` took 69s
and 75s, even though a background step fills the npm cache first.
> - The loopback registry in `scripts/public-install-registry.mjs` sends
packages that it does not serve to npmjs with a 302 redirect. It escaped
scoped names with `encodeURIComponent` (`%40scope%2Fname`). npm requests
and caches the same metadata as `@scope%2fname`, and npm keys cached
metadata by URL.
> - Thus the consumer install could not use the scoped metadata that the
warm-up had just put in the cache. It downloaded and parsed all 291
scoped packuments again (about 285 MB of full packument JSON) inside the
4-vCPU container.
> - This pull request makes the redirect use the same escaping as npm.
> - The benefit is a shorter Build check on every pull request. The
installed bytes do not change.

## Linked Issues or Issue Description

**Describe the bug**
The consumer install in the public-install verifier does not use cached
scoped package metadata. The warm-up step
(`scripts/warm-public-install-npm-cache.mjs`) caches this metadata
through direct npm requests.

**Steps to reproduce**
1. Run `PAPERCLIP_PUBLIC_INSTALL_NPM_CACHE=/tmp/c node
scripts/warm-public-install-npm-cache.mjs`.
2. Install the same dependency set through the loopback registry with
`npm_config_cache=/tmp/c` and `--loglevel http`.
3. Count the `(cache miss)` lines.

**Expected behavior**
No cache misses for metadata that the warm-up already fetched.

**Actual behavior**
291 cache misses. All of them are scoped packages that are requested as
`https://registry.npmjs.org/%40scope%2Fname`. The cache holds them as
`https://registry.npmjs.org/@scope%2fname`.

Related: #16513 changes when the verifier waits for the warm-up. This
pull request does not overlap with it, and the two changes add together.

## What Changed

- `scripts/public-install-registry.mjs`: the 302 redirect for packages
that the registry does not serve now uses npm's own escaping
(`name.replace('/', '%2f')`, the same as npm-package-arg `escapedName`).
The `packageName` check already allows only characters that need no
other escaping.
- `scripts/__tests__/public-install-registry.test.mjs`: the test now
expects `@other%2fpkg` for a scoped redirect. A new assertion covers an
unscoped redirect.

## Verification

- `node --test scripts/__tests__/public-install-registry.test.mjs`: 3 of
3 pass. This includes the real npm install test that compares installed
bytes from file installs and registry installs.
- Local replay (Linux, npm 11.12.1). I filled a fresh cache with the
warm-up script, then installed the same 62-dependency set (627 packages)
through the registry:
  - Before: 291 cache misses, 28.3s.
  - After: 0 cache misses, 19.8s.
- I also ran the real `scripts/public-install-registry.mjs` with a
packed root package that depends on the full public dependency set.
Result: 2 misses (only the loopback package and its tarball, as
expected) and 1438 cache hits.
- To compare on CI, look at `consumer_install_ms` in the
`paperclip.public-npm-install.timing.v1` line of the Build log.
Baseline: 68822 ms (run 39254079691) and 75175 ms (run 39251304886).

## Risks

- Low risk. The redirect target is the same npmjs document. Only the URL
escaping changes. npm still checks every tarball against the integrity
in the packument.
- If the npm cache holds stale scoped metadata, npm revalidates it the
same way as it does for direct requests (`cache-control: max-age=300`).
- The expected time saving comes from a local measurement. The real
Build saving depends on the network and CPU of the runner. I expect
about 10–20s per Build run. The CI timing line will confirm the number.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code as a Paperclip agent with tool use (shell, file edit). I did the
root-cause analysis, the local replays and the code change with this
model.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com>
Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-04 12:10:00 -08:00
ff7e7b4963 fix(claude-local): make sandbox tool commands the first OOM victims (#16844)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Hosted `claude_local` runs execute in a sandbox. The sandbox has a
fixed memory limit (4 GiB in the observed sandboxes).
> - In that sandbox, every process has the same `oom_score_adj`. When a
Bash tool command uses most of the memory, the kernel kills the largest
process. That can be the Claude Code process.
> - When Claude Code is killed, `claude-agent-acp` reports "The
connection to Claude was lost." and the whole run fails. Server logs
show `Claude Agent process died: Claude Code process terminated by
signal SIGKILL` for each failure.
> - A failed tool command is recoverable. A dead agent is not.
> - This pull request makes Claude's tool commands the first
out-of-memory victims in managed sandboxes.
> - The benefit is that a memory-heavy command fails with exit 137, and
the agent continues the run.

## Linked Issues or Issue Description

Refs #16366 (diagnostic PR for the same failure). Related: #14509
protects the host control plane from local agent processes. That PR does
not cover a sandboxed agent and its own tool commands.

**What happened?**

Hosted `claude_local` runs fail with `acpx_turn_failed` and "The
connection to Claude was lost." while one `execute` (Bash) tool call is
pending. The pending commands are memory-heavy, for example `tsc
--noEmit` with a 3.3 GB heap, a monorepo `pnpm install`, or a type check
plus `vitest`. At the same second, the server log shows `Claude Agent
process died: Claude Code process terminated by signal SIGKILL`. Over 10
days, all 12 such lines on one staging stack were `SIGKILL`, and they
match the failed runs. The sandbox memory cgroup has `memory.max` 4 GiB
and `memory.oom.group` 0, so the kernel kills one process. Every process
has `oom_score_adj` 0, so the kernel picks the largest one.

**Expected behavior**

Under memory pressure, the kernel kills the tool command. Claude gets a
failed command (exit 137) and continues the run.

**Steps to reproduce**

1. Start a `claude_local` run on a sandbox target with a 4 GiB memory
limit.
2. Ask the agent to run a memory-heavy command in a large repository,
for example a full `tsc --noEmit` and a test run together.
3. The kernel kills the Claude Code process, and the run fails with "The
connection to Claude was lost."

**Agent adapter(s) involved**

Claude Code (`claude_local`, ACP lane and CLI lane, managed sandbox
targets).

## What Changed

- `claude-config.ts`: add `CLAUDE_TOOL_SHELL_PREFIX_SCRIPT`. Claude Code
calls a `CLAUDE_CODE_SHELL_PREFIX` with one argument, the full command
string. The script raises its own `oom_score_adj` to 1000 (no privilege
is necessary to raise it). Then it runs the command string in the shell
that Claude Code selects: `CLAUDE_CODE_SHELL`, then `SHELL`, when either
is an executable bash or zsh, else zsh or bash from `PATH` (bash first
when `SHELL` names bash).
- `buildRemoteClaudeConfigMaterializationCommand` can install the script
(umask 077, write to `.partial`, `chmod 700`, `mv`). The install runs
last in the existing materialization command, so it adds no sandbox
round trip. It prints a marker only when the file is in place.
- `materializeRemoteClaudeConfig` returns `toolShellPrefixInstalled`. It
is true only when the command exits 0 and prints the marker.
- ACP lane (`acp.ts`) and CLI lane (`execute.ts`): set
`CLAUDE_CODE_SHELL_PREFIX` to the installed script only when the install
is confirmed. A prefix path that does not exist would fail every Bash
tool call, so a failed install keeps the previous behavior. An operator
`CLAUDE_CODE_SHELL_PREFIX` in the adapter env is kept, and no script is
installed.
- `docs/adapters/claude-local.md`: add "Sandbox memory limits".

## Verification

- `npx vitest run src/server/claude-config.test.ts` in
`packages/adapters/claude-local`: 10 pass, 6 new. The new tests install
the script with the real materialization command through `/bin/sh` and
check:
- the file content, mode 0700, the marker, and that no `.partial` file
remains;
- the command runs with `oom_score_adj` 1000 while the caller keeps its
own score (Linux only);
- shell selection: a valid bash `SHELL`, an invalid `CLAUDE_CODE_SHELL`
that falls back to `SHELL`, and a `SHELL` of `/bin/sh` that falls back
to bash from `PATH`;
  - quoting and the exit status of the command are kept;
  - an operator prefix wins, and no marker means no install.
- `acp.test.ts`: the managed-config test now checks that
`CLAUDE_CODE_SHELL_PREFIX` names an executable 0700 file next to the
config dir. A new test checks that an operator prefix is kept and no
script is installed. Both use the local sandbox runner, which runs the
setup commands for real.
- `tsc --noEmit -p packages/adapters/claude-local` is clean.
- Two tests fail in my sandbox both with and without this change: `uses
the same default model in ACP startup and session identity` and
`reselects the full assignment and bootstrap guidance after a failed
resume`. They also fail on clean `master` in the same environment, so I
think they depend on that environment. CI covers them.
- Real Claude Code check (Claude Code 2.1.286): with
`CLAUDE_CODE_SHELL_PREFIX` set to the script, a Bash tool call printed
`oom_score_adj` 1000 and ran in zsh (the selected shell), and its parent
was the `claude` process at score 0. Without a prefix file that exists,
every Bash tool call fails with exit 127. This is why the env is set
only after a confirmed install.
- Real OOM check in a 4 GiB Daytona sandbox (`memory.oom.group` 0). A
stand-in "agent" process held 1.2 GB at score 0, and a tool command
started 14 processes of 250 MB each:
- without the prefix, the kernel killed the stand-in agent (SIGKILL,
137; `oom_kill` +1);
- through the installed prefix, the stand-in agent stayed alive, and the
kernel killed 5 tool processes (`oom_kill` +5).
- Assumption that could invalidate the design: Claude Code passes one
argument (the command string) to the prefix and honors the variable
through `claude-agent-acp`. I checked the argument shape with a logging
prefix, and `claude-agent-acp` 0.73.0 spawns the CLI with `{
...process.env }`.

## Risks

- Every Bash tool command and hook in a managed sandbox now runs through
one more `sh` process and an `exec`. The cost is a few milliseconds for
each command.
- If a future Claude Code release changes how it calls
`CLAUDE_CODE_SHELL_PREFIX`, Bash tool calls can fail. The tests above
and the documented variable limit this risk. Removing the variable
restores the old behavior.
- The inner shell is not a login shell. Claude Code sources its own
shell snapshot inside the command string, so the environment stays the
same. If Claude Code runs without a snapshot, it uses a login shell.
Only exported variables then pass through to the inner shell.
- If the selected shell differs from the one that Claude Code chose (for
example, a custom `CLAUDE_CODE_SHELL` that is not bash or zsh), the
script uses the same fallback order as Claude Code.
- This does not help when the Claude Code process itself uses most of
the memory. The local (non-sandbox) lanes do not change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with tool use
and code execution. It read the repository and the Claude Code CLI
behavior, ran the OOM experiment, edited the code, and ran the tests.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [ ] I have run tests locally and they pass (two tests fail on clean
`master` in my sandbox too; see Verification)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-04 12:09:54 -08:00
Nicky LeachandClaude Opus 5.5 e107a75755 fix(native): decide session settlement from live state (#18992)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native Runner sessions keep one provider conversation across task
turns, so a later run attaches to a session that an earlier run settled.
> - Three gates decide if a session is settled: Core's journal check,
the Runner's attach check, and the Runner's drain of provider messages
after a turn.
> - Each gate kept a list of harmless events and refused anything else.
Every new harmless event became an outage.
> - An example: Codex replays a settled turn's usage on `thread/resume`,
so `run.attach` refused every session that restarted after its idle
timeout.
> - Events after a terminal describe the past. Refusing the session
cannot undo them, and it hides them.
> - This pull request asks a different question: is anything live or
waiting right now? Past events are recorded and no longer block.
> - The benefit is that harmless events stop breaking session
continuity, with no event list for Core and the Runner to keep in step.

## Linked Issues or Issue Description

No issue exists. I searched open and closed pull requests and issues for
`run.attach requires a settled Codex provider session` and
`prior_owner_terminal_state_indeterminate`. Related work: #18890 added
one more accepted event to Core's list. This change replaces that list.
#18991 handles late usage and is independent of this change.

**What happened?**

On a test deployment, 11 runs failed in 21 days when `run.attach`
refused with `requires a settled Codex provider session with no pending
events`. In all 11, the four pending events were two startup audit
events, `session.resumed`, and a `codex_resume_usage_snapshot`
diagnostic. The diagnostic is what the attach check refused. The retry
then failed with `prior_owner_terminal_state_indeterminate` in 6 of
them, which hid the first failure.

**Expected behavior**

A session with nothing live or waiting accepts the next run. If
attachment fails, the run reports the real cause.

**Steps to reproduce**

1. Finish a Codex turn on a native Runner session.
2. Wait for the idle timeout so that the provider process stops.
3. Start the next run. Codex replays the usage for the settled turn on
`thread/resume`.
4. The attach fails.
`durable_backend_attaches_after_a_settled_resume_usage_snapshot`
reproduces this on the unfixed code.

**Paperclip version or commit**

Observed on `8fd917aedf`. The same check is on `master` before this
change.

**Deployment mode**

Cloud deployment, with a native Runner in a plugin sandbox environment.

## What Changed

- **Runner attach:** `attach_run` refuses only for live or waiting
state: no thread, a closed session, a session that needs reconciliation,
an active or ambiguous turn, an active goal, a pending tool call, or an
unanswered provider request. The error now names the reason.
- **Pending events:** they no longer block attachment. They stay in the
retained queue, and the Runner commits them under the prior authority
before it rotates. `session.resumed` is still discarded because it is
superseded.
- **Runner drain:** `drain_completed_turn_tail_for_warm_attachment` no
longer needs a list of known messages. It still fails on a message that
names another turn, on `turn/started`, on a provider request, and on a
process exit.
- **Core:** `isSettledRemoteWarmControlState` accepts any authenticated
event after the terminal unless it starts work (`turn.started`,
`turn.accepted`, `run.attached`, `tool.execution.started`,
`semantic_tool.input`, `runtime_request.created`), or a goal snapshot or
update shows an active goal or work in progress. All identity, source
and protocol checks are unchanged.
- **Visibility:** the next run's log names the event types that were
recorded after the previous run finished.
- **Real cause:** a same-run retry after a failed attachment now fails
with `prior_attach_failed: <original failure>`, instead of
`prior_owner_terminal_state_indeterminate`.
- **Docs:** added "Session settlement after a run" to
`doc/execution-semantics.md`.

## Verification

- `cd packages/paperclip-runner/runner && cargo test --locked -p
paperclip-runner-core --test codex_provider`: 102 passed, 2 ignored.
`--lib`: 478 passed, 2 ignored. `--test durable_recovery`: 3 passed.
- **Failure before the fix:**
`durable_backend_attaches_after_a_settled_resume_usage_snapshot` failed
on the unfixed code with the same error as production.
- `cd server && pnpm exec vitest run
src/services/native-runtime/remote-warm-session-recovery.test.ts
src/services/native-runtime/native-session-executor.test.ts -t "settled
remote warm control evidence|recovers a blocked|automatically recovers
only a proven settled local session|retained|warm"`: 476 passed, 680
skipped by the filter.
- `pnpm exec vitest run
src/services/native-runtime/native-session-resume.test.ts`: 57 passed.
- `npx tsc --noEmit -p .` in `server` reports no errors.
- **Real data, read-only:** I decrypted the 11 failed runs' stored
events on a test deployment and printed only event types, field names,
and thread and turn IDs. All 11 show the same four events.
- **Audit of live state:** I read the Runner's provider state and
request tracking to build the attach check. It is a code audit. I have
not proved that every kind of live work sets a flag. The Runner holds
active turns, ambiguous starts, tool calls and provider requests in
state today.
- **Not covered:** the wiring that throws `prior_attach_failed` has unit
coverage for its message only, not an end-to-end test.
- **Review:** Greptile Apex scored 4/5, then 5/5 at the current head.
- **Not run locally:** the full test suite. CI provides that coverage.
- **Not verified live:** no live run has used this change.

## Risks

- **Fail-open for unknown live work:** an event type that starts work,
but is not in the short list and sets no state flag, would now pass.
This is the main tradeoff. The previous list failed closed, but it
caused real outages.
- **More events reach the journal after a terminal.** The usage replay
diagnostic is now delivered rather than refused. These events are
informational, and they are named in the next run's log.
- **A failed attachment is not retried on the same run.** Some refusals
could clear on their own, such as a turn that finishes, and some attach
errors are not about live work, such as a failed provider restore. The
run now reports the cause, and it needs another run to continue. On a
test deployment, the next successful run after each failure used a new
provider thread and a new execution workspace. A later change could add
same-run retry.
- **Rollout order:** deploy Core before the Runner. An older Core would
refuse the new events that a new Runner commits after a terminal.
- **Migrations:** this change has no migration.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be reworked.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), through Claude Code,
with tool use and code execution. A human directed the design and
approved the scope.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-04 12:04:29 -08:00
DottaandPaperclip 25b5a072cc fix(recovery): let messages continue recovered preparation (#19026)
## Thinking Path

> - Paperclip must preserve work while allowing people to continue
failed tasks.
> - A preparation failure can retain an ID from an earlier provider
session.
> - Workspace recovery and host settlement already prove that the failed
attempt never dispatched a provider.
> - The preparation classifier also required that inherited session ID
to be absent.
> - That extra condition kept new messages, saved messages, and Retry
behind a resolved recovery hold.
> - Classify execution from the existing execution evidence and admit
one fresh turn through normal gates.

## Linked Issues or Issue Description

Refs #18908, #19004, and #19021. The latter two address historical Retry
ancestry and empty object logs. This repair addresses a current source
with an inherited session and ordinary user messages; it needs neither
ancestry nor an object log. ROADMAP.md and related open PRs were
checked.

**What happened?**

Sending a message to a failed task saved it in the queue, but did not
start another run. The failed native preparation had recovered files and
durable host settlement, plus `sessionIdBefore` from prior work. The
classifier treated that input as execution evidence and kept the
recovery hold.

**Expected behavior**

Admit the new user turn once the existing saved-work and stop checks
pass. Preserve previous failures and files. Show the current run instead
of the old error, and consume messages when execution starts.

**Steps to reproduce**

Fail Runner preparation before runtime selection. Preserve its remote
workspace. Retain an inherited `sessionIdBefore`. Send an ordinary
message or click Retry. The three new HTTP regressions accept the
request but admit zero successors before the repair.

**Paperclip version or commit**

Base `3bd52a9fa0`.

**Deployment mode**

Authenticated hosted Runner. Regression verification uses macOS arm64,
Node 26.4.0, real PostgreSQL, HTTP routes, and real temporary workspace
archives. The external sandbox transport and native provider boundary
are controlled in the execution fixture.

## What Changed

- Stop requiring the inherited session input to be absent from otherwise
verified failed preparation.
- Cover new messages, previously saved messages, Retry, request replay,
one successor, recovered files, and current task presentation.
- Exercise the existing preparation safety controls with inherited
context, including a new output-session refusal.
- Exercise native execution and finalization with the inherited session
input.
- Document the continuation rule.

## Verification

- RED: all three new HTTP cases admitted zero successors before the
production change.
- All three final HTTP cases pass. The 41 preparation safety cases,
including the new output-session refusal, pass with inherited context.
- The native execution fixture passes with inherited context. It reads
restored files, writes the continuation result, runs one provider turn,
persists the accepted result, empties the queue, and resolves the
notice.
- Initial positive assertions expected an empty queue while the fixture
deliberately held the agent's slot, then expected a null optional wait
field. The final assertions use the actual queue contract and retain
execution and presentation checks.
- `pnpm build`, `node scripts/typecheck-low-memory.mjs`, module
boundaries, and `git diff --check` pass locally. The final rebased head
`56d2ef26` passes the four HTTP/native journey cases. All 634 tests in
the four complete owning recovery, Retry and continuation suites pass.
Current-head full CI passes: 53 successful checks and three expected
skips, with no failed or pending checks. The full local command has the
documented timeout below.
- The full local command passed its first three chat shards, then
stopped in shard four on a 15-second timeout in `answers each finding
thread in its own run instead of one unattestable batch` (267 other
shard cases passed). The exact unchanged test passes alone (about nine
seconds of test work) and all three full CI chat groups pass on this
head. It uses a custom adapter, so the changed native-preparation proof
is inapplicable. The timeout cause remains unknown; the full local
command is not claimed green and its later groups were not reached.
- Initial CI attempt: all other checks passed, but server shard 2/12
failed on the unchanged `idle-completed-history.test.ts` owned HTTP
runtime test. The ten-second readiness probe returned ECONNREFUSED. The
source and runtime have no diff from the base and this test does not use
failed-preparation admission. The exact local case passes; the
underlying readiness cause remains unknown. Logs and annotations are
retained at [the failed
job](https://github.com/paperclipai/paperclip/actions/runs/41897539206/job/126811385017).
The single failed-job retry passed on the same head, including the
dependent verify aggregate. This proves recovery of that check; the
original readiness cause remains unknown.
- Greptile reviewed the exact head `56d2ef26` at 5/5 with no actionable
findings and zero unresolved threads. Hosted recovery remains unverified
until the repair reaches the serving revision.

## Risks

The risk is mistaking earlier context for a stopped current execution.
Current provider session, process, controller, native
input/result/output, accounting, exact lease stop, workspace
epoch/generation/writer, actor, assignment, pause and approval checks
remain enforced. No migration, new polling, API, credentials or
execution policy is added.

## Model Used

OpenAI GPT-6 through Codex, using reasoning, code editing, shell tools,
and browser diagnostics. The exact serving snapshot and context window
are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (634 owning-suite cases and
final-head journey cases; full-local timeout is documented above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-04 13:54:38 -06:00
Devin FoleyandPaperclip e65c68726b Require published migrators before publishing Core images (#19023)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Its release pipeline publishes Core images and exact-source database
migrators.
> - A migrator run can pass admission but skip all build and publication
work.
> - The image observer could stop checking after provisional build
admission.
> - The final image gate checked source proof without checking migrator
publication.
> - This pull request requires the exact migrator publisher to finish
before image publication.
> - Consumers can then use the published image with its matching
migration artifact.

## Linked Issues or Issue Description

**What happened?**

The [migrator run for
e968131](https://github.com/paperclipai/paperclip/actions/runs/41868122685)
passed its admission job but skipped build and publish. Its [Docker
run](https://github.com/paperclipai/paperclip/actions/runs/41868122627)
still published an image. The exact-source migrator artifact was absent.

**Expected behavior**

An image consumer restarts a strictly deferred producer. It waits for
the exact migrator publication before it publishes the image. A
successful admission job alone does not prove publication.

**Steps to reproduce**

1. Let a migrator run defer work while the image consumer starts.
2. Make the producer appear admitted before its build later becomes
skipped.
3. Complete the source verification job.
4. Observe that the old image gate can accept source proof without a
successful migrator publisher.

The regression fixtures model this state. The real producer records
establish the missing publication; the exact transient API response seen
by the original observer is not retained.

**Paperclip version or commit**

The observed source was `e968131efa7e63d321f3bcca8d47a62513c2efb4`. This
patch is based on current master.

No duplicate fix was found in open migrator PRs. PR #18406 concerns npm
canary scheduling and does not provide this image publication gate.

## What Changed

- Require the successful canonical migrator `publish` job for the exact
source and latest attempt. Check complete job pages and re-read the run
after the jobs.
- Continue observing deferred producers until publication, even after a
build appeared admitted. Only a physical runner proves build admission.
- Check migrator publication again in the final read-only
`--check-proof` image gate. Failed producers still require inspection
and are not retried automatically.
- Add regression coverage for deferred recovery, skipped and stale
publication, attempt races, and the final CLI gate. Update release
documentation.

## Verification

- All 57 current-head check contexts are successful or expected skips on
`daed1c756a9fc02b8c15128b96af4da86d447e3f`; both required checks (`ci /
verify`, `ci / e2e`) passed. Greptile is 5/5 with no review threads, and
the branch has no merge conflicts. GitHub master rules require CODEOWNER
review before merging.
- The first CI attempt failed serialized server shard 2/9:
`heartbeat-process-recovery.test.ts` timed out after three seconds
waiting for an adapter execution to start (1 failed, 411 passed).
[Failed
job](https://github.com/paperclipai/paperclip/actions/runs/41894941958/job/126802368555).
This PR changes no server or package files. The exact timeout cause is
not established; this failure is retained and is not described as
infrastructure noise.
- The same first attempt also failed general server shard 6/12:
`heartbeat-stale-queue-invalidation.test.ts` waited five seconds for an
adapter execution after native-owner completion; the run remained
running with no error (1 failed, 2,627 passed, 3 skipped). [Failed
job](https://github.com/paperclipai/paperclip/actions/runs/41894941958/job/126802369329).
Its cause is also not established. The first-attempt failures remain
part of this record.
- The single controlled failed-jobs rerun completed successfully on
attempt 2 at the identical commit. Both previously failed heartbeat
shards and the aggregate verifier passed. This demonstrates recovery on
rerun; the underlying first-attempt failure causes are not established.
[Successful
rerun](https://github.com/paperclipai/paperclip/actions/runs/41894941958/attempts/2).
- Node 24: `node --test scripts/cloud-source-verification.test.mjs
scripts/standard-image-contract.test.mjs` passed: 62 tests, 0 failures,
after rebasing on current master.
- An explicitly gated read-only live probe rejected the real skipped
producer for `e968131` and accepted the published producer for
`afbc5a885b52e96df10c1c6e643bea31b51caac8`.
- `git diff --check` and Gitleaks passed before push.
- A broader run that also included
`scripts/__tests__/release-verify-workflow.test.mjs` failed when local
fixture creation and package installation hit `ENOSPC`. Those failures
remain unverified locally; the focused source and image tests passed
separately, and CI Typecheck + Release Registry subsequently passed.
- Full workspace typecheck, Vitest and app builds were not run locally.
Disk capacity does not permit installing the full workspace in this
separate checkout. No application TypeScript or package manifest
changed. The complete CI run subsequently passed, including typecheck,
build, general and serialized tests, release checks and e2e.
- No live Slack posts, workflow dispatches, producer reruns, deployments
or provider mutations.

## Risks

- Image publication now waits for the exact migrator publisher. A failed
publisher blocks the image and reports a publication failure. Existing
source failure cancellation still applies.
- This adds bounded Actions reads to the existing source observer. The
observer keeps its existing deadline. The final publication runner only
reads evidence and cannot restart producers.
- The publisher job name is part of this gate. A workflow rename must
update the gate.
- No database schema, migration content, image contents, credentials or
application behavior change. Revert this PR to restore the prior
publication gate.

## Model Used

- OpenAI `gpt-6.1-sol`, xhigh reasoning effort, through Codex in T3
Code. Source inspection, tool use and code execution. Context window
size is not exposed by the harness.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass — focused tests pass;
broader local ENOSPC failures are recorded above
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-04 11:51:42 -08:00
Devin FoleyandClaude Opus 5.5 09f9d691cf fix(server): keep the root failure code when the execution deadline fails a run (#19028)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The execution-control watchdog fails a run when its control deadline
passes and no owner finished it.
> - The watchdog replaces the run's error code with
`execution_finalization_deadline_exceeded`.
> - The run-failure report reads the root cause from
`resultJson.originalFailureCode`. The watchdog did not write that field.
> - So every watchdog failure goes to Sentry with the same fingerprint,
whatever its first failure was.
> - This pull request keeps the earlier failure code as
`originalFailureCode`, as the native finalizer already does.
> - The benefit is that Sentry groups watchdog failures by their real
first failure, so each cause can be found and fixed.

## Linked Issues or Issue Description

**What happened?**

One hosted Sentry group collects all
`execution_finalization_deadline_exceeded` run failures. It has 17
events from many drivers (Codex app server, OpenCode server, ACPX
runtime). In one recent event, the run first failed with
`native_execution_ownership_unverified`. The watchdog then failed it at
the deadline. The recovery action evidence kept `originalFailureCode:
native_execution_ownership_unverified`, but the run row and the Sentry
event did not. The event had no `root_error_code` tag, and its
fingerprint was only `[execution_finalization_deadline_exceeded,
paperclip_runner]`.

**Expected behavior**

When the watchdog replaces an earlier, more specific error code, the run
keeps that code as `resultJson.originalFailureCode`. The run-failure
report then sends `root_error_code` and adds it to the fingerprint
(`server/src/sentry.ts`).

**Steps to reproduce**

1. Create a running run with `errorCode:
native_execution_ownership_unverified` and an expired
`executionControlDeadlineAt`.
2. Run `reconcileAbandonedExecutionControl`.
3. The run fails with `execution_finalization_deadline_exceeded`, and
the earlier code is lost. The captured event has no
`diagnostics.execution.originalFailureCode`.

**Paperclip version or commit**

`master` at `89703c3ff7de`.

Refs #14110, a similar fix for the finalization sweep. A search found no
other open PR for this path. Draft #18672 changes legacy deferred wakes,
which is a different path.

## What Changed

- `execution-control-reconciliation.ts`: when the watchdog fails a run,
it writes the earlier code (`coordinator.failureCode ?? run.errorCode`)
to `resultJson.originalFailureCode`. This is the same value that it
already writes to the recovery action evidence.
- The watchdog keeps the other `resultJson` fields.
- The watchdog does not replace a root cause that is already recorded.
- The watchdog does not record the deadline code as its own root cause.
- Two regression tests in `execution-control-reconciliation.test.ts`.

## Verification

- The new test "keeps the run's prior failure code as the reported root
cause" fails on unchanged `master`. The run's `resultJson` has no
`originalFailureCode`. With the fix, the test passes.
- `pnpm exec vitest run
src/services/execution-control-reconciliation.test.ts
src/__tests__/execution-control-work.test.ts
src/__tests__/run-failure-sentry.test.ts src/__tests__/sentry.test.ts`:
111 passed, 5 skipped.
- `pnpm exec vitest run src/services/task-retry.test.ts
src/services/heartbeat/recovery.test.ts
src/services/native-runtime/native-safe-replacement.test.ts
src/services/workspace-recovery.test.ts`: 437 passed. These suites use
this watchdog cause.
- `node scripts/typecheck-layers.mjs` in `server/`: passed.
- Not run locally: the full `pnpm -r typecheck` (the sandbox has no Rust
toolchain for the runner crate), `pnpm test:run` and `pnpm build`. CI
runs these.

## Risks

- Low risk. The change adds one `resultJson` key only when the watchdog
moves a run to `failed`, and only when a different earlier code exists.
- Other readers of `resultJson.originalFailureCode`
(`native-run-projection.ts`, `idle-sleep-safety.ts`) only check
committed or succeeded runs. A watchdog failure is neither, so their
results do not change.
- `isPendingNativeRetry` reads only `nativePhase`, so reporting is not
suppressed.
- Sentry groups will split by root cause after deployment. New events
will go to new groups, and the existing group will get fewer events.
This is the intended change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution
in a Paperclip agent sandbox.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-04 11:51:36 -08:00
Dotta 58ff18c731 fix(evals): validate timeline actor clauses and exit outcomes 2026-11-04 13:46:54 -06:00
DottaandPaperclip 3a283fbc9b fix(runtime): preserve failed sources with older warm supervisors (#19024)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A volume-free web server needs durable recovery data for retained
sessions.
> - An old failed follow-up can retain its snapshot while the warm
supervisor still holds the preceding successful turn.
> - The migration refuses to treat those different snapshots as the same
execution authority.
> - That refusal also blocks the tenant even when the failed source can
be preserved as an archive.
> - This pull request archives that narrow case after it verifies the
older successful snapshot and all scope bindings.
> - The failed run keeps its history and files while valid controller
recovery keeps its existing proof requirements.

## Linked Issues or Issue Description

Refs #18959, which added the existing failed-controller archive path. No
duplicate PR for this case was found.

**What happened?**

Offline controller migration stops with `stale_supervisor_snapshot` for
an old failed warm follow-up. Its local supervisor still contains the
prior successful turn. Read-only checks found the same database snapshot
and supervisor bytes in the live source and the isolated copy. The
mismatch predates the rehearsal.

**Expected behavior**

Preserve the failed run's database state and available local files in
the existing durable archive. Only archive when the older supervisor
matches a completed successful run in the same company, agent, task,
native session and Runner instance. Keep current execution and
controller authority checks intact.

**Steps to reproduce**

Create a successful native turn and a later failed warm follow-up. Keep
the successful snapshot in the supervisor file and the failed snapshot
in the database. Give the failed source local controller bytes but no
durable controller checkpoint. Run the offline migration. The new
regression fails with `stale_supervisor_snapshot` before this change.

## What Changed

- Verify the older supervisor against scoped successful history and the
same provider thread.
- Use the existing encrypted archive owner for that failed legacy
source.
- Preserve snapshots, run status, ownership, queued work and the
session's last-run pointer.
- Keep foreign, changed, active, successful and already-durable sources
outside this path.
- Document the preservation and recovery boundary.

## Verification

- New positive regression: red on the original owner, green after the
change.
- Native migration and controller recovery: 27 checks passed. Final
migration checks also cover refusal to replace a durable checkpoint.
- Storage failure blocks publication. Repeated migration is safe. The
archive restores the exact controller and supervisor bytes after the old
home is deleted.
- The archived failed source does not gain executable recovery
authority.
- Local server typecheck is blocked by reused workspace declarations for
accounting, bootstrap and ingress contracts. No diagnostic names either
modified owner or test file. Full local typecheck, repo-wide tests and
build are not claimed; clean CI is required before merge. The exact-head
CI has now passed typecheck, build, policy, Runner checks, browser E2E,
and all server test shards.
- Rehearsal verification against the copied historical source remains
required. Full tenant migration and hosted deployment qualification
remain separate gates.

## Risks

The archive preserves evidence for an already failed source. It does not
certify that source as resumable. A prior snapshot must match exactly,
and the normal archive owner checks the terminal failed binding before
it publishes. Existing durable checkpoints remain authoritative. A
missing or changed proof still blocks migration.

## Model Used

OpenAI Codex, exact model ID `gpt-6.1-sol`, with reasoning, tool use and
code execution. The host does not expose context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have described the issue in-PR following the bug template
- [x] I have not referenced internal or instance-local Paperclip issues
or links
- [x] My branch name describes the change and contains no internal
ticket id or instance-derived details
- [x] I have run the relevant tests locally; retained failures and
limits are listed above
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
canary/v2026.1104.0-canary.12
2026-11-04 13:40:53 -06:00
Dotta 1b5ebfaf7b fix(evals): share timeline takeover provenance guards 2026-11-04 13:40:12 -06:00
Dotta 3a9c3fa8cc fix(evals): recognize successor takeover in timeline context 2026-11-04 13:40:12 -06:00
Dotta 2569e44944 fix(evals): accept recorded passive timeline handoffs 2026-11-04 13:40:12 -06:00
Dotta b20730d647 fix(server): clarify failed-action chat completion feedback 2026-11-04 13:36:09 -06:00
Devin FoleyandPaperclip cf0ab4f7f4 chore: enable boxd plugin release publishing (#19020)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Sandbox provider plugins let those agents run in remote
environments.
> - The boxd provider exists, but its release entry excludes it from CI
publishing.
> - npm now has the `@paperclipai/plugin-boxd@0.0.0` bootstrap
placeholder.
> - This pull request enables CI publishing and documents npm
installation.
> - A pre-existing OpenCode fixture blocked CI because it qualified a
group-writable host Node binary.
> - This also gives that test its own Linux Node copy with strict
permissions.
> - Operators can install a real release after CI publishes it.

## Linked Issues or Issue Description

Related implementation:
https://github.com/paperclipai/paperclip/pull/17284.
No matching public issue or duplicate release-enrollment PR was found.

**What existing behavior does this improve?**

Publishing and installing the existing boxd sandbox provider.

**Subsystem affected**

`packages/plugins` and the npm release manifest.

**Current behavior**

`publishFromCi` is false. npm has only the `0.0.0` placeholder. The
README says the package is not published and requires a local-path
install.

**Proposed behavior**

Include boxd in CI releases. Document stable installation, explicit
published versions, and the placeholder limit.

**Reason and benefit**

Operators can install the provider through the existing npm plugin
installer after a real release ships.

**Breaking changes**

None. This changes release enrollment and documentation only.

**Additional context**

This follows the existing plugin and sandbox roadmap items. It adds no
new runtime capability.

## What Changed

- Set `publishFromCi: true` for `@paperclipai/plugin-boxd` in the
release manifest.
- Replace the README's unpublished-package notice with npm installation
guidance.
- Explain that the `0.0.0` placeholder has no plugin functionality and
that `latest` can point to it until the first stable release.
- Keep the local development instructions available.
- Fix the existing ingress-retry test to use an owned Linux Node copy
and explicit executable permissions. This follows the existing OpenCode
transport fixture. It does not change production launch checks.

## Verification

- Passed: `node --test scripts/release-package-map.test.mjs
scripts/check-release-package-bootstrap.test.mjs
scripts/bootstrap-npm-package.test.mjs` (31 tests).
- Passed: `node scripts/release-package-map.mjs check`.
- Passed: `PAPERCLIP_RELEASE_BOOTSTRAP_BASE_SHA=origin/master node
scripts/check-release-package-bootstrap.mjs
scripts/release-package-manifest.json
packages/plugins/sandbox-providers/boxd/README.md`.
- Passed: `git diff --check`.
- Confirmed that npm resolves `@paperclipai/plugin-boxd` to version
`0.0.0`.
- Full workspace typecheck, test suite, and build were not run locally.
CI checks remain required before merge.
- Initial CI failure: [native Runner
job](https://github.com/paperclipai/paperclip/actions/runs/41889969053/job/126785636239),
OpenCode ingress retry, `qualified OpenCode proxy command must not be
group- or world-writable`. The failing fixture is byte-identical to the
base commit `3a2249dd576`. It passed `process.execPath` to
qualification, while other OpenCode tests already use a private Linux
Node copy.
- Local focused run: `pnpm exec vitest run
server/src/services/native-runtime/native-ingress-bootstrap-retry.integration.test.ts`
built the real Rust Runner and passed the Codex case. The OpenCode case
stopped at `runner_remote_provider_artifact_incompatible: OpenCode proxy
path belongs to the controller host` on macOS (`/Users/...`). This does
not exercise the new Linux-only copy.
- An unchanged-base OpenCode comparison was attempted with `-t
"reacquires opencode ingress"`, but PostgreSQL setup failed with
`ENOSPC` before the case ran. The local baseline comparison is therefore
inconclusive. Task-owned Rust build outputs were cleaned to recover disk
space.
- At this stage Linux verification was pending. The successful final run
is recorded below.
- No new tests were needed. The existing release-tool tests cover
manifest enrollment, dependency ordering, and bootstrap checks.

- Current-head CI interruption: [workspaces-a shard
1/2](https://github.com/paperclipai/paperclip/actions/runs/41893808561/job/126799012853)
was cancelled after the runner received a shutdown signal. The log shows
no assertion failure. The job rerun request was rejected with HTTP 403
because the workflow was still running. The subsequent fixture
correction starts a new full CI run and supersedes that pending rerun.

- CI at `7fffa315`: [native Runner
job](https://github.com/paperclipai/paperclip/actions/runs/41893808561/job/126799011711)
passed executable permission checks, then failed because Node tried to
resolve `/memfd:paperclip-verified-launch (deleted)`. The private
executable was named `opencode-node`, which bypassed Runner's Node
interpreter recognition. The fixture now names its copy `node`, matching
the existing transport fixture and selecting the CommonJS descriptor
loader.
- Passed a local smoke test of the production CommonJS descriptor loader
using an open, unlinked script file. This is macOS evidence only; the
successful final Linux run below provides the complete ingress retry
evidence.

- Final verification on `bcacac542aa0a1b641b26c099c1e9fc4a1653ec4`: all
54 reported checks passed; three unrelated checks were skipped. [Native
Runner integration
job](https://github.com/paperclipai/paperclip/actions/runs/41895787712/job/126805844773)
passed, including the OpenCode ingress retry regression. The fresh full
run also passed the workspace shard that was interrupted on the previous
attempt. Greptile reports 5/5 for this exact head with no unresolved
review threads. Earlier failures and local verification limits remain
recorded above.

## Risks

- The maintainer confirmed that npm trusted publishing is configured for
this package. The first real CI publish will verify OIDC authentication.
- Merging enables publication in the next successful canary run. An
installation that uses `latest` can still resolve to the placeholder
until the first stable release includes this change.

## Model Used

OpenAI Codex, exact model ID `gpt-6.1-sol`, with xhigh reasoning effort,
repository tool use, and shell execution. The runtime does not expose
the context window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-04 11:36:04 -08:00
DottaandPaperclip 5c378d5837 test(evals): version all credential destination consumers
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-04 13:27:42 -06:00
DottaandPaperclip 8da511c603 fix(evals): preserve credential destination clauses
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-04 13:27:02 -06:00
Dotta 89703c3ff7 fix(ui): place pinned sidebar pages above More (#18915)
## Thinking Path

> - Paperclip helps people manage AI agents and their work.
> - The Coordinator sidebar gives people access to workspace pages.
> - People can pin pages from the More menu.
> - The sidebar puts these links below More in a separate Pinned
section.
> - This change puts pinned links above More in the main navigation.
> - People can reach their saved pages without an extra section heading.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

The placement of pinned pages in the Coordinator sidebar.

**Subsystem affected**

ui/ — React board UI.

**Current behavior**

Pinned pages appear below More with a Pinned heading and extra space.

**Proposed behavior**

Pinned pages appear after Needs you and before More. They use the same
row spacing as the main links.

**Reason and benefit**

Saved pages belong with the main navigation. This placement keeps them
visible above the menu that contains the other pages.

**Breaking changes**

The Pinned section and its heading are removed. Saved pin data, link
order, routes, and pin controls use the existing code.

## What Changed

- Move pinned links into the primary navigation above More.
- Remove the separate section, heading, and extra space.
- Correct the feature guide to state that pinned pages appear above
More.
- Update sidebar tests to check placement, saved order, account changes,
restore, and pin removal.

## Verification

- `pnpm exec vitest run ui/src/components/CoordinatorSidebar.test.tsx`:
15 tests pass.
- `pnpm check:token-gates`: pass.
- `node scripts/check-maps.mjs`: pass after the guide correction.
- `pnpm --filter @paperclipai/ui typecheck`: pass.
- Browser checks pass in Chromium against the production Coordinator
sidebar in Storybook with fixture data. Screenshots cover saved pins and
pinning from More. No page errors occurred. This does not test a live
API.
- The full workspace typecheck and build stop because this host has no
Rust `cargo` command.
- The first UI build was killed and the browser crashed while checks ran
together. The host recorded OOM kills at its 8 GB limit. The browser
check then passed after the full-suite process was stopped. The UI
production build passed after the Storybook server was stopped.
- `NODE_OPTIONS=--max-old-space-size=4096 pnpm --filter @paperclipai/ui
build`: pass.
- `pnpm test:run`: stopped during the first chat integration shard after
memory exhaustion affected other checks. It has no full-suite result.
The CI run passed the test lanes on the latest head.

- [CI
run](https://github.com/paperclipai/paperclip/actions/runs/41768881396)
passed on `a836fc1320c363e0fbb5591312c6726026b6f151`. The required `ci /
verify` and `ci / e2e` checks pass. Build, typecheck, Rust checks, test
lanes, and all eight browser shards pass.
- Greptile: 5/5 on this head. The documentation finding is fixed and its
thread is resolved.
- The additional Paperclip review is paused and queued. It is not a
required branch check.

## Risks

Low risk. This change moves existing links. It does not change storage
or API behavior. The tests check expanded and collapsed rendering, the
empty state, stored order, unknown routes, user changes, and changes
from another browser tab.

## Model Used

OpenAI GPT-6 through the Codex agent. The session supports reasoning,
tool use, code edits, and command execution. The runtime does not expose
a more specific model ID or context window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-11-04 13:10:32 -06:00
DottaandPaperclip d180ad8c77 fix(heartbeat): retry chat execution authorization contention (#19017)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A task Retry must use the current task and chat authority.
> - The request and queue paths already retain a Retry through brief
lock contention.
> - Execution checks that authority twice, including after native input
preparation.
> - A busy task row or chat identity could fail the admitted run at
either check.
> - This pull request retries only the rolled-back authorization
transaction.
> - The same run can proceed after the lock clears. The final dispatch
gate still rejects changed ownership.

## Linked Issues or Issue Description

**What happened?**

A failed chat task can accept a normal Retry and then fail during
execution setup when another PostgreSQL transaction holds the task row
or chat identity lock. The late check can fail after native input and
workspace preparation. The exception can then enter the workspace
preservation path.

**Expected behavior**

Keep the same admitted run and its frozen input. Retry a recognized,
rolled-back authorization conflict within the existing bound. Recheck
current authority before each attempt. Reject Stop, paused agents,
revoked chat access, or a changed task owner before the execution
callback.

**Steps to reproduce**

1. Admit a normal failed-chat task Retry with the existing PostgreSQL
fixture.
2. On a second connection, hold the task row or the chat identity
advisory lock when execution authorization starts.
3. Release the lock after the real authorizer reports contention.
4. Observe that the original code fails the run before the authorized
callback. Repeat at the late native check after input preparation.

**Paperclip version or commit**

Reproduced on master `039e0dad2c51c80251e1302a19c5566e039c357a`. The
branch is rebased on `8f06bd4e1`.

**Deployment mode**

Source checkout with real PostgreSQL and the local HTTP Retry route.
Provider and chat transport responses use controlled test fixtures.

Refs: #18895 and #18896 cover request and queue admission. This change
covers execution authorization. #18898 and #18989 cover predecessor
cleanup, a separate boundary. The roadmap and public PR search show no
duplicate execution-authorization repair.

## What Changed

- Recognize the existing typed chat-presentation contention error in the
bounded admission retry helper. Keep its existing PostgreSQL
lock-unavailable handling.
- Use that helper at both execution authorization checks. Each attempt
checks cancellation and the applicable legacy controller lease, then
opens a fresh transaction. The loop contains no provider or workspace
callback.
- Include failed-chat Retry in the existing final dispatch ownership
gate. This preserves current task ownership and native recovery owner
checks after a delay.
- Add real PostgreSQL contention, late prepared-input, Stop, pause,
revocation, owner replacement, native completion, and ordinary legacy
completion controls. Document the retry and dispatch boundaries.

## Verification

- The unchanged base failed real task-row and chat-identity contention
cases. A separate late native case failed after frozen input
preparation. A safety control then proved that a replaced task owner
could still reach the backend; the final dispatch gate fixes that path.
- All 11 PostgreSQL cases and all four helper controls pass. The tests
keep exact frozen model and trust input, reject changed authority, and
do not replay a provider callback.
- A real local HTTP Retry reaches the native executor, one controlled
provider turn, one accepted result, a committed finalizer, and one
published chat reply. Reprocessing the canonical request does not start
another turn.
- A separate real HTTP ordinary legacy Retry runs one controlled adapter
callback, completes its task through the normal service, and does not
replay the canonical request.
- Independent review found no material issue in the two production
files. It checked transaction scope, lock order, current ownership,
native lease expiry, and import cycles.
- The full workspace typecheck and full `pnpm build` pass on this head.
Full `pnpm test:run` passed all 1,916 chat integration cases. Its
broader server group reported the existing workspace-runtime case
`returns a bounded structured conflict and exposes only same-company
workspace references` as failed after 46 ms. The unchanged focused case
and the complete 179-case file both pass on rerun. The active broad run
was then stopped for this finite handoff; its child exited 130 and the
command exited 1. The detailed assertion was buffered and was not
retained, so its exact cause remains unknown. This is a partial local
run, not a full local pass; later local groups did not run. All three CI
chat shards pass. [Greptile reviewed this exact head at 5/5 with no
findings](https://github.com/paperclipai/paperclip/pull/19017#issuecomment-6638521119).
- Retained setup and fixture failures are not counted as product
regressions or passes. An incomplete copied Copilot runtime failed the
first typecheck; the supported installer repaired that isolated
dependency, then typecheck passed. Early acceptance fixtures used a
wrong publication field or did not complete their task; the final
controls use the real field and completion service.
- The first CI run failed [workspace shard
1](https://github.com/paperclipai/paperclip/actions/runs/41885475363/job/126769239992)
on child-server startup and a five-second dense ZIP test deadline.
[Server shard
2](https://github.com/paperclipai/paperclip/actions/runs/41885475363/job/126769239908)
failed an existing local runtime readiness probe. All 18 cases in the
two CLI files and the runtime case then passed unchanged locally on this
head. The CLI files also match an earlier independently verified head
byte for byte. One failed-job rerun was requested. Its [server
job](https://github.com/paperclipai/paperclip/actions/runs/41885475363/job/126777070120)
was cancelled after the runner received a shutdown signal. It produced
no test verdict. After that workflow ended, only the interrupted server
job and its required dependents were rescheduled once. The [complete
workspace rerun
passed](https://github.com/paperclipai/paperclip/actions/runs/41885475363/job/126777070125).
The [rescheduled server
job](https://github.com/paperclipai/paperclip/actions/runs/41885475363/job/126781960353)
and its aggregate passed. The [complete current-head workflow is
green](https://github.com/paperclipai/paperclip/actions/runs/41885475363).
All required current-head CI and security gates pass, including the
Ready-triggered scans; three conditional checks are skipped. The
complete unchanged idle-history file also passes all 75 cases locally.
No timeout or assertion changed; all attempts remain part of this
result.
- These local controls do not claim hosted-provider or Linux sandbox
acceptance. The original task-row failure did not retain a SQLSTATE; the
PostgreSQL reproduction proves the local gap without assigning that
original cause.

## Risks

- The existing bound allows 50 backoffs of 100 ms. It limits backoff
time, not total database or lease wall time.
- Native ownership is checked at the final dispatch gate. The
per-attempt controller lease assertion applies to legacy execution.
- A persistent lock or changed authority still fails closed. Unknown
prior provider effects and accepted work retain their existing guards.
- No migration or model, account, trust, credential, or permission
policy change is included.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, tool use, code editing, and
PostgreSQL verification. The host does not expose the exact serving
model ID or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass within the stated scope:
relevant regression and acceptance controls, and the complete chat,
idle-history, and workspace-runtime files. The broad local run remains
partial as recorded above.
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-04 13:07:50 -06:00
DottaandPaperclip 3bd52a9fa0 test(evals): pin task-history discovery and governed handoff repairs (#18995)
## Thinking Path

> - Paperclip must complete work through ordinary human and agent
controls.
> - The paired timeline attempt omitted a real human handoff, although
the other ten checks passed.
> - The agent used the always-exposed active-task history tool as if its
comments were a status audit.
> - The tool now states its actual scope and directs chronology and
handoff questions to the existing discovery and API tools.
> - A separate approved-tool attempt preserved permission and write
boundaries, but paired Markdown emphasis caused a false outcome
rejection.
> - The reviewed oracle normalizes that formatting before the existing
outcome checks and retains balanced negatives.
> - This PR pins those corrections with the already reviewed personal
secure-handoff admission repair for one new representative batch.

## Linked Issues or Issue Description

Refs #18962, #18968, #18971, #18972, #18977. Follows merged #18976.

The original paired results remain FAILED. Result 4412 omitted the
actual human handoff; this is a real answer gap, not a date-oracle
excuse. Result 4411 correctly handled the first rejection with no writes
or Trust action and rejected agent permission, but the observer misread
paired inline emphasis in the truthful outcome. The personal-secret
result 4378 remains FAILED at admission; secure save and worker proof
were not reached. No prior grade is rewritten and no new native pass is
claimed.

## What Changed

- Pin immutable candidate `249b487a2d2451763913038cc009f7090580d1f1` on
`codex/chat-catalog-human-answer-admission-candidate-20261104`.
- Extend exact prior source `b1119630003785c9e26e14c763859125642ad756`
with only the accepted timeline-discovery and formatting corrections: 12
source, test and generated files, with 72 additions and ten removals.
- Use the canonical `get_task_history` description in the actual exposed
semantic catalog. It reads only bounded active-task comments; calling
`get_task` does not change that active scope. It is not a status or
assignment audit. Progress and handoff questions must discover the
company timeline or issue activity, then use the existing API call.
- Regenerate semantic catalog and protocol projections on this frozen
candidate. Preserve all non-description fields, schemas and authority
boundaries. Do not copy newer master catalog or document-schema changes.
- Normalize paired inline emphasis before approved-replay outcome
checks. Declare only that oracle as app-review v6. Keep timeline v5,
personal v5 and company v4.
- Retain the reviewed personal completion-context, addressed-human,
optional-note and rendered-placeholder safety controls. Actual private
save, owner isolation, audit, authorized worker delivery and native
continuation remain required.
- Update only the existing manifest, expected pin and descriptor
snapshot in this infrastructure PR. Preserve all 244 identities,
original requests, run/human counts, clocks, model, launcher bytes,
privacy limits and qualification holds.

## Verification

- The combined candidate passed 542 existing owning controls across 19
suites, including catalog contracts, consistency and 242 lazy
declarations with no fixture callbacks. The manifest retains two legacy
tasks, for 244 identities.
- Candidate Runner checks passed: 15 semantic consumer tests and 12
catalog tests. The owning Runner TypeScript compiler passed after the
existing eval-kernel prerequisite built. The first compile reported
missing ignored eval-kernel output; the repair used candidate-owned
ignored dependency links and output only. No repository type or
acceptance rule was changed.
- All three generated checks passed: semantic-action catalog, semantic
contracts and protocol manifest. Deep comparisons prove each generated
catalog differs only in `get_task_history.description`. The seeded
fixture changes only its catalog hash, and the protocol manifest changes
only the hash of that fixture. The compiled canonical catalog hash
matches the fixture. The candidate import conflict added only
`getTaskHistoryAction`; no unrelated newer document-schema import was
copied.
- Both owning fixes cleared fresh exact-head Greptile 5/5 with no
actionable threads before this publication: timeline
`7f6ba9686b5edba005a49cd15c0d0b52248178dc` and approved replay
`0015a79c1e0a3af5292edf4a61aa9bab8f381899`. The personal prerequisite
`789bec74a0f43011723fff45596ef9d084bfaba7` also cleared exact-head 5/5
and all 12 threads before its earlier integration.
- All 131 existing pin, TestOps contract/API, adapter, reconciliation
and streaming controls passed. The descriptor snapshot matches the
actual candidate bytes. The only new declaration change from the prior
candidate is app-review v5 to v6. Whitespace checks passed.
- The SDK source guard reports 506 unchanged inputs, runtime pins
unchanged, registry
`efa70621e9fb42aa185f7ee616a1a726aa3eccb367e67e28817cedf26ef2ade8`. No
SDK registry or runtime certificate renewal occurred. The existing
reporting-lock guard returned `already-reviewed` with lock
`60f4c06bca47af3ab9d74281f2ef2e1ff559d566b862ba51c2fbe6c8e5bac35b`.
- Historical CI results remain recorded. Head 73ea had a server-5
cancellation with exact AWS Spot-reclamation proof. Head 16134 had 44
successful jobs, three failed leaves and dependent verify failure, plus
one selector skip. Two failed hosts had exact Spot termination proof;
server 11 had two real deadline PostgreSQL failures. After an initial
libzstd setup block, the unchanged four-test suite passed locally on
exact merge `70e68bff5fb89251f769671ce9ab949ab5bf7fc9` in 33.69 seconds
with its original deadlines and assertions. This does not establish the
CI cause. One recovery POST returned HTTP403; zero replications were
accepted.
- Prior head c95 finished with 41 successful jobs, six failures, one
cancellation and one selector skip. Four leaves had exact
Spot-termination proof; three of their execution logs were unavailable.
Server 11 instead had a real `live-recovery-work` next-deadline
assertion failure, and server 4 had a real workspace-health
connection-refused failure. Dependent verify failed. Windows passed. No
replay or claimed test fix erased those results; their two assertion
causes remain unresolved.
- Exact head `1e89df928f7ccb43bf7af3b814f2c2207d75883a` now has
successful normal CI run 41888521175 (attempt 2: 48 successful jobs and
one selector skip) and successful Windows run 41888521015. The first
attempt retained 46 successes, the failed Typecheck leaf and failed
dependent verify, plus one selector skip. Typecheck reported runner
shutdown without a TypeScript diagnostic; its Spot request reported no
capacity, while the later EC2 termination reason was
Client.UserInitiatedShutdown. One authorized unchanged-head targeted
recovery reran only Typecheck and dependent verify; both passed. This
does not claim a code fix or establish the cleanup caller. All other
successful results were carried forward, not fresh remeasurements.
- Current exact-head Greptile and Paperclip assessments are both 5/5
with no actionable findings; Paperclip check conclusion is neutral, but
its final assessment completed and states Ship. Security checks passed
and no review thread is unresolved. Full repository build/typecheck/test
were not repeated locally for this three-file metadata change. A fresh
actual-serving source, reporting-lock and selection guard remains
required after a normal merge and before a new Submit.
- No model call, eval dispatch, secret mutation, private archive
download, fleet change or merge was performed by this update. The single
targeted CI recovery is recorded above. Separate core PRs remain
unmerged.

## Risks

The discovery description may change tool choice; its effect on the
missing handoff remains unproved until a new native attempt. Formatting
normalization can affect outcome admission, so existing positive and
negative semantics remain required. The personal secure-entry and worker
phases are still unproved. The next representative set is timeline,
approved-tool replay and personal secret. Company selection remains
gated on personal secure-entry and worker proof. The uniform ten-case
regression remains pending, and the full 244-case catalog remains
unqualified.

## Model Used

OpenAI `gpt-6.1-sol` through Codex. Capabilities used: source analysis,
isolated mechanical integration and local code execution. The host does
not expose the context-window limit. The evaluated model remains
OpenCode/OpenRouter `deepseek/deepseek-v4.1-flash-1017`.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-04 13:00:37 -06:00
DottaandPaperclip 249b487a2d test(e2e): read paired emphasis before review outcome checks
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-04 12:29:25 -06:00
Dotta 4713327d22 fix(runner): clarify active task history discovery scope 2026-11-04 12:29:25 -06:00
DottaandPaperclip 3a2249dd57 fix(runner): retry proven ingress bootstrap failures (#18981)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native execution must retain its admitted model and execution
authority during recovery.
> - A sandbox provider can reject Runner endpoint acquisition before the
Runner gets a credential.
> - The old error path loses the structured HTTP status and leaves that
startup failure ambiguous.
> - A zero counter in an old checkpoint cannot prove that no credential
was issued.
> - This pull request adds a versioned producer proof and carries closed
provider status through the existing RPC.
> - A proved transient startup failure uses the existing retry budget. A
terminal proved-unused run can settle its accounting.

## Linked Issues or Issue Description

Refs #18018 and #18934. Those changes restore existing checkpoints and
ready retained environments. This change covers endpoint acquisition
before Runner dispatch. A search of related public PRs found no
duplicate implementation.

**What happened?**

A provider HTTP 502 during Runner endpoint acquisition becomes an
untyped worker error. Native recovery treats it as ambiguous and stops
without retrying. The run's native usage receipt also stays pending
after startup ends without model work.

**Expected behavior**

Retry a structured transient endpoint failure only when the complete
current controller checkpoint proves that this exact run never issued a
Runner credential. Preserve authentication, configuration, Stop,
ownership and provider-progress gates. Settle zero usage only under the
existing terminal accounting owner.

**Steps to reproduce**

1. Use the controlled endpoint in the Daytona ingress integration test
to return HTTP 502.
2. Register the actual native Runner driver and persist its bootstrap
journal through the encrypted checkpoint writer.
3. Observe the old terminal ambiguous disposition instead of bounded
startup recovery.

**Paperclip version or commit**

Reproduced on the pre-change master source. The original regression used
the production native driver, a real HTTP fault, a real durable control
plane and local Postgres. The new contract additionally tests the pinned
Daytona SDK and real worker RPC.

## What Changed

- The controller owns a v2 persisted state with retained origin and
credential history. Current readers still accept v1. Older writers
reject v2. Restore and checkpoint decoding do not add missing history.
- The proof validates the complete bootstrap journal, all six identity
fields and every delivery, credential and progress control. The server
rechecks its fingerprint under the current controller claim before retry
or accounting changes.
- Daytona and the SDK carry a closed acquisition phase and HTTP status
through the existing ingress RPC. Both public error wrappers omit
response bodies and causes, including Pino cause serialization.
- Only structured HTTP 429 or 5xx ingress failures can use the existing
three-attempt startup budget. HTTP 401/403, configuration and unknown
errors do not gain retry authority.
- A terminal proved-unused run wakes existing accounting and records
zero provider work. It does not create a semantic result, prove physical
termination or release an unknown process/workspace hold.
- The normal automatic second attempt rechecks its exact claim, frozen
execution and matching complete local/encrypted journal before admitting
absent remote Runner state. It keeps the original pending bootstrap
command IDs. A proved unused Codex home can rematerialize sealed skills.
Full raw local state is checked before streaming-event or policy
projection.
- Normal saved-state readers and the existing E2E journal evidence
matcher accept the closed v1/v2 set. The separate legacy no-launch
witness is unchanged. Technical docs describe the contracts.

## Verification

Current head: `cd38f70409cdb8664e14bfbf4efa208f468a0419` (final delta
adds only fixture artifact preparation). Production files are unchanged
from the independently reviewed `93c8a33520` source. Full repository
typecheck and build passed at `96417acb58`; the affected server
typecheck also passes after the final two-line fixture delta. Focused
protocol, raw-proof, real retry, SDK and RPC acceptance pass on this
source. Earlier broad suite counts are listed with their limits below:

- Durable Runner protocol: 187 tests.
- Native executor, driver, startup/accounting and redaction: 1,211
tests, including real Postgres and encrypted checkpoint persistence.
- Normal saved-state restore/projection compatibility: 21 focused cases.
- SDK: 209 tests.
- Standalone Daytona: 544 tests; 8 opt-in live tests skipped.
- Explicit standalone `pnpm -C
packages/plugins/sandbox-providers/daytona exec vitest run --config
vitest.sdk-ingress.config.ts`: 6 cases with real pinned SDK 0.203.0 and
worker RPC ingress acceptance. These include lookup 502/401/403/400,
private-preview 502 after successful lookup/refresh, missing credentials
and private-response exclusion.
- E2E journal evidence: 4 cases across both supported formats, with
wrong ownership, missing stimulus and unknown-schema controls.
- The exact historical controller source rejects a new v2 checkpoint at
decode and restore. A v1 control is accepted. This verifies the
downgrade fence with the actual older parser.
- Startup controls cover issued and pruned credentials, old/truncated
journals, provider progress, exhausted retry accounting, and
checkpoint/Stop/controller-claim races. Race cases retain the current
execution and pending usage.
- New exact admission controls: 19 cases through encrypted persistence
and cold restore. They cover current/stale/expired claims, Stop, pause,
model/trust/account changes, prior progress, issued/pruned credentials,
adopted/old/truncated state, local/canonical fingerprint mismatch,
hidden delivery, unsupported work and company scope. Two additional
red/green cases prove that raw local stream progress and missing event
policy are rejected before lossy checkpoint projection. No case mutates
run, claim, result or accounting state.
- Actual native executor, scheduler and Rust Runner retry: 2 cases.
Codex uses its fake app server. OpenCode uses the real proxy and fake
OpenCode HTTP server with the frozen DeepSeek model and allow permission
configuration. Each path launches no Runner on attempt 1, starts one
Runner and provider on attempt 2, reuses both original command IDs,
commits one accepted result, validates the persisted local workspace,
records nonzero synthetic native usage and settles accounting once. No
zero-work accounting flag survives.

The controlled SDK and protocol tests use local source, isolated
PostgreSQL, a private HTTPS preview and fake provider servers on macOS
arm64 with Node 24.22.2. They create no hosted sandbox and make no model
call. The root workspace intentionally omits standalone provider
dependencies; the explicit SDK target fails if its pinned dependency is
absent and has no conditional skip. They do not qualify hosted rollout
or repair an old unmarked checkpoint. The first full local typecheck
found missing checkout SDK links from an install with scripts disabled;
the repository link helper repaired that setup.

Current-head Linux CI typecheck and build pass. Independent final review
checked 12 source scopes and six raw-proof controls with no remaining
finding. Fresh Greptile review is 5/5 with no unresolved thread.

The full Runner CI suite found two older control-peer test doubles
missing the new proof method. Both now return no proof, with all 27
permission, ordering and cancellation assertions retained and passing.
The new Cargo-dependent native retry suite uses the existing Rust-cached
CI lane. All 39 lane coverage controls pass and prove complete,
duplicate-free suite coverage. The first full test run stopped at an
admitted chat Retry polling failure. CI also found a stale
retained-allocation fixture, now repaired on master by #18984, and a
canceled runner job. A real SDK suite initially relied on a dependency
absent from the root workspace; the explicit standalone target now owns
that acceptance.

The separate Product E2E support run had 12 failures in four untouched
files. All 12 reproduce on the original clean base with independent
dependencies, compiled base source and verified workspace resolution.
These concern workflow expectations, cleanup timeout arguments, a
profile snapshot and operating receipts. They are not passing
acceptance.

The later full local repository test command stopped at a 15-second
timeout in the unchanged GitHub pre-allocation retry `controller` case.
That exact case passed in isolation with unchanged assertions. The broad
Runner run initially omitted `test:typescript:prep` and lacked debug
fake-provider binaries. Required preparation then passed 370 + 44
package contract cases (11 skipped); the prepared full package run
reported 4,932 passed, 9 failed and 13 skipped. All failed families then
passed with unchanged assertions in isolation: 8 ACPX controller tests
and 2 credential controls. The paired original-base ACPX control also
passed. The full-run failures remain recorded; they are not passing
full-suite evidence.

The first Rust-cached native CI run found a missing generated OpenCode
proxy in a fresh checkout. The fixture now runs the existing verified
provider-entrypoint builder. Removing the generated proxy reproduces the
exact ENOENT before the fix; both real retry cases pass after the fix
from the same absent-artifact state.

All current-head CI gates pass, including the aggregate verification
check. Linux native CI passes all 10 suites: 177 tests pass and 14
opt-in cases skip. Both real provider retry cases pass. The full Runner
package passes 297 suites and 4,950 tests, with 1 suite and 10 tests
skipped. The CLI workspace groups pass 4,762 and 286 tests. The first CI
attempt had Runner shutdowns in the Runner package, Canary and three
general server jobs. One unchanged CLI shard also hit child-server
startup and dense ZIP deadlines. Both CLI files passed unchanged in
isolation: 18 tests. One bounded rerun of unsuccessful jobs and their
dependents passed. The original failures remain recorded.

## Risks

- Older controllers reject newly written v2 state on downgrade. Current
controllers retain v1 restore support, but old or unmarked state cannot
gain this new proof.
- An older plugin that omits structured ingress evidence remains held.
The server and provider must use the new producer contract for this
recovery path. The Docker `cloud` target builds the SDK, server/Runner
and standalone Daytona producer from one source tree. The default
`production` target does not build the standalone producer. Hosted
rollout must confirm the new image and active image-owned plugin worker;
a registry version alone does not prove its source. An npm-installed
producer needs an explicit matching artifact upgrade.
- This prevents future proved startup failures. It does not release
historical accounting based on an old checkpoint, change retry budgets,
replace a model or weaken sandbox isolation.
- Zero model usage is separate from process termination, saved files and
resource release. Those owners keep their existing gates.

## Model Used

OpenAI Codex with tool use and code execution. This runtime does not
expose the exact backend model ID or context window to the agent.
Read-only independent agents reviewed checkpoint compatibility and
SDK/RPC error boundaries.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (affected suites; full
local run limits are recorded above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
canary/v2026.1104.0-canary.11
2026-11-04 12:16:51 -06:00
Devin FoleyandClaude Opus 5.5 6a45ea479e fix(heartbeat): retry failed legacy lease renewals until the deadline (#17851)
<!-- entire-trail-link-start -->
https://entire.com/gh/paperclipai/-/trails/52/changes/paperclip/52
<!-- entire-trail-link-end -->

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A legacy agent run keeps a durable controller lease. The server
renews the lease every 10 s. The lease lasts 60 s.
> - The lease watchdog stops the run when it loses the lease, because
another controller can then take the run.
> - #17032 moved renewal to a private pool. It also retries a renewal
that hits its statement timeout. But a renewal that fails with any other
error still stops the run at once.
> - Staging still shows this failure after #17032. A legacy run made 186
good renewals over 31 minutes. Then one renewal failed with an error
23.9 s after the last good renewal, and the watchdog stopped the run
immediately (`controllerLeaseLossCause=renewal_error`). The lease still
had about 36 s left.
> - This pull request retries a failed renewal until the lease deadline,
the same as a timed-out renewal.
> - The benefit is that one dropped database connection no longer
cancels a healthy run. The 60 s lease deadline still bounds the retries,
so the ownership guarantee does not change.

## Linked Issues or Issue Description

Refs #17032
Refs #16084

**What happened?**
A legacy agent run on staging was cancelled mid-turn. The run was
healthy. The cancellation evidence was
`cancellationSource=controller_lease` and
`controllerLeaseLossCause=renewal_error`, with
`controllerLeaseRenewalCount=186` and
`controllerLeaseSinceRenewalMs=23913`. The release already contained
#17032.

**Expected behavior**
One failed lease renewal must not stop a run while the lease is still
valid. The watchdog must try the renewal again. It must stop the run
only when the lease deadline passes or when the database rejects the
renewal.

**Steps to reproduce**
1. Start a legacy run so that `watchLegacyControllerLease` watches it.
2. Make one renewal query fail with a connection error, for example
terminate its Postgres backend with `pg_terminate_backend`.
3. See that the watchdog aborts the run immediately with
`renewal_error`, although the lease has not expired.

## What Changed

- `watchLegacyControllerLease` retries a renewal that fails with an
error. It waits 1 s between attempts
(`LEGACY_CONTROLLER_RENEWAL_RETRY_DELAY_MS`). The existing lease
deadline stops the retries.
- A retried renewal is safe. The UPDATE is idempotent, and it still
requires this controller's boot ID and an unexpired lease. A rejected
renewal (no row updated) still stops the run at once with
`renewal_rejected`.
- If the deadline passes while the attempts still fail with an error,
the cause stays `renewal_error`. A statement timeout counts as a stall,
so it stays `renewal_deadline`, as before. A good renewal clears the
error state.
- `deadlineLateMs` is now recorded whenever the deadline ends the lease,
also for `renewal_error`.
- Tests:
- New embedded-Postgres test: it terminates the renewal's own backend
while the renewal waits on the run row. The renewal retries on a new
connection and keeps the lease. Without the fix, `assertOwned` rejects.
- New unit tests: one failed attempt is retried and the lease is kept; a
stall after a recovered error is still `renewal_deadline`.
- The existing `renewal_error` test now checks that a renewal that
always fails is retried and is reported as `renewal_error` at the
deadline, with its timings.
- `doc/run-log-events.md` describes the retry and the updated
`renewal_error` and `deadlineLateMs` meaning.

## Verification

- `server/src/services/legacy-controller-lease.test.ts` with embedded
Postgres: 23 tests pass.
- With the original `legacy-controller-lease.ts`, 4 of these tests fail.
One of them is the real-Postgres test that terminates the renewal
connection.
- `server/src/services/heartbeat/run-completion.test.ts` and the
`run-cancellation` tests: 35 tests pass.
- A typecheck of the changed files reports 0 errors. CI runs the full
typecheck.

## Risks

- Low risk. The change is limited to the legacy lease watchdog.
- A run with a permanently failing renewal now stops at the lease
deadline (at most 60 s after the last good renewal), not at the first
error. Another controller cannot take the run before the lease expires
in the database, so ownership stays safe.
- A non-transient error (for example a code bug in the query) is retried
once per second until the deadline. The cause label is still
`renewal_error`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, code search, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-04 10:15:08 -08:00
DottaandPaperclip b111963000 fix(evals): use completed personal setup context for acknowledgements
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-04 11:57:25 -06:00
DottaandPaperclip 8f06bd4e17 fix(e2e): retain workspace packages in rolling app copies (#18982)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A rolling deployment must preserve running native agents and their
work.
> - The source test lane starts separate copies of the compiled app.
> - The development Runner entry imports source from the workspace
packages directory.
> - That directory was absent from each copied app, so its supervisor
could not start.
> - This pull request preserves the package link beside the existing CLI
link.
> - The test lane can then exercise real controller replacement.

## Linked Issues or Issue Description

**What happened?**

The source rolling-deployment supervisor failed with a missing Runner
module before starting a paid agent. The copied app had its compiled
server, dependencies and CLI link, but lacked the workspace package path
used by the development Runner entry.

**Expected behavior**

The copied app resolves its development dependencies and reaches
controller replacement.

**Steps to reproduce**

1. Build the server and UI from source with the development Runner
entry.
2. Run `tests/runner-e2e/controller-replacement-smoke.ts` with
disposable PostgreSQL and S3.
3. The old copy fails to resolve the Runner package. The fixed copy
completes the supervisor smoke.

No duplicate PR or issue was found in the exact-path search. This is a
test-harness fix within the existing rolling-deployment lane.

## What Changed

- Link the checkout's `packages/` directory into each copied source app.
- Document the shared dependency links and the limits of source-lane
evidence.

## Verification

- The missing-module failure was reproduced before the link was added.
- The same supervisor smoke passed after the fix and confirmed cleanup.
- The ten-check non-agent rolling smoke passed with disposable
PostgreSQL and S3.
- A fresh strict Codex rolling browser case passed. It kept the same
running process, native session and sandbox lease, acquired a new
controller generation, finalized the workspace, and continued from the
saved project.
- An earlier paid attempt failed the strict tool oracle because the
agent's own soak script omitted the argument delimiter for
hyphen-prefixed input. That failure remains recorded; the verifier and
task contract were unchanged for the passing attempt.
- Server TypeScript compilation passed in the source test checkout. Full
repository typecheck, tests and build are required in CI for this clean
PR head; they have not been repeated locally in this dependency-free
checkout.

- Exact head `d8addaa7723af5a013489e5bb6331dd1021dc6be`: all CI checks
passed or were intentionally skipped; Greptile 5/5 with no review
threads. Earlier cancelled jobs and the old recovery-fixture failure
remain retained; rebasing onto its existing master fix passed cleanly.

## Risks

Low production risk: only source test setup and its documentation
change. Each compiled app still has a separate cache and home. Its
workspace dependencies are shared, so this does not qualify a released
container or Railway deployment.

## Model Used

OpenAI Codex, exact model ID `gpt-6.1-sol`, with reasoning, tool use and
code execution. The host does not expose the context window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues OR (b) described the
issue in-PR following the relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links
- [x] My branch name describes the change and contains no internal
ticket id or instance-derived details
- [x] I have run tests locally and they pass, with the earlier material
failure retained above
- [x] I have added or updated tests where applicable: the existing
supervisor and live rolling browser lane exercise this path
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-04 11:56:13 -06:00
Devin FoleyandPaperclip edb9d29942 fix(recovery): preserve Retry authority and reopen cross-task cases (#18997)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Tasks keep execution identity and saved workspace files separate
from control-plane updates.
> - A scanner can have permission to reopen a case while its run cannot
restore that case's files.
> - Retry can also follow a preparation failure that has no
recovery-action record.
> - Both paths can accept valid work but leave the target unable to run.
> - This change restores files under the target executor and saves
admitted Retry authority on the canonical receipt.
> - Existing workspace, user, source and successor checks still apply.

## Linked Issues or Issue Description

**What happened?**

An authorized agent updates another closed task. The route tries to
restore the target workspace under the caller's run identity and returns
`403 Task workspace run identity is unavailable`.

A separate failure occurs when a normal board Retry follows a
pre-provider preparation failure. Admission has no recovery-action row
on which to save continuation authority. Dispatch then fails with
`continuation_user_authorization_missing`.

**Expected behavior**

Accept the authorized task update. Restore its terminal workspace when
the target's admitted executor starts. An admitted Retry must retain its
exact dispatch authority even when the predecessor has no
recovery-action record.

**Steps to reproduce**

1. Complete a task with an isolated workspace. From a different running
task, use an authorized comment or PATCH with `resume: true` on that
closed task.
2. Separately, fail native preparation before provider work starts, with
no recovery-action row. Submit board Retry, admit its successor, and
build the execution continuation.
3. Before this change, the first path returns 403 and the second rejects
its admitted authority.

**Paperclip version or commit**

Reproduced against `e968131efa7e63d321f3bcca8d47a62513c2efb4`. Rebased
onto `afbc5a885b5`.

**Deployment mode**

Server-managed native task execution. Tests use real PostgreSQL and
temporary workspace files.

Related PRs: #16927 covers terminal workspace races. #17916 retains
exact native export sources. Neither covers these route and Retry
receipt failures. A GitHub search found no duplicate. ROADMAP.md was
checked; this is a recovery bug fix.

## What Changed

- Defer workspace restoration only for authorized cross-task comments
and updates from a live run. Checkout keeps its existing workspace
identity checks.
- Restore the target's selected terminal workspace under its own
admitted run. Preserve its saved files and existing source checks. Leave
manually archived workspaces unchanged.
- Store server-created continuation authority on the canonical admitted
Retry receipt when no source recovery-action record exists. Existing
action records remain authoritative. Read the fallback only after
matching the company, task, agent, user, source, generation and
successor.
- Add route and PostgreSQL regression tests. Document both paths.

## Verification

- Red: both cross-task route tests returned the observed 403 before the
route fix.
- Red: the target executor could not reuse its archived terminal
workspace before the restoration fix.
- Red: an admitted preparation Retry with no recovery-action row failed
with `continuation_user_authorization_missing` before the receipt fix.
- Green: 419 recovery and continuation tests pass, plus 183 focused
route, environment and Retry tests. The assembled heartbeat test starts
the agent in the same terminal worktree and verifies both its unpushed
commit and uncommitted file bytes. It fails before the restoration fix.
- Negative cases reject wrong Retry runs, users, source runs,
generations, unadmitted receipts and missing authority. The original
failed run stays unchanged. Workspace tests verify saved file bytes.
- Full `pnpm build`, `pnpm -r typecheck`, `pnpm check:module-boundaries`
and `git diff --check` pass.
- The full suite found an existing assembled chat authority regression:
clearing a recovery-action record could be bypassed by the new fallback.
The fix now writes the fallback only when that source has no action
record. That assembled regression passes with the original test
unchanged.
- Full `pnpm test:run` is running again. Its first attempt also hit an
unrelated ancestor skill-root lookup for a missing Slack skill. The
rerun uses a temporary untracked link to the repository's own bundled
skills. This link is excluded from the PR. Full-suite completion and
hosted CI are pending.
- Hosted CI has multiple runner interruptions. E2e shard 1/8, server
shards 3/12, 8/12 and 12/12, and workspaces-a shard 1/2 show runner
shutdown signals. E2e shard 5/8 and server shards 1/12 and 2/12 have
GitHub annotations reporting lost runner communication; their logs
return 404. The `ci / verify` assertion failed because the general-test
matrix was cancelled. GitHub rejected job and failed-workflow reruns
while this workflow is still running. Rerun the interrupted checks after
it settles.
- Greptile reviewed exact head
`a1e3929aedb1ee3bfcd001f4ebb1510106bf4070` at 2026-11-04 17:37 UTC: 5/5,
with no new blocking findings and no unresolved review threads.

## Risks

- A deferred workspace restore can still fail at target admission. The
run keeps the existing workspace failure path; this change does not
bypass file recovery or provider stop checks.
- Retry receipts now store server-created authority. Existing action
receipts remain valid. No schema migration is required.
- Local regression tests establish the software behavior. The new
revision has not been deployed to a hosted instance.

## Model Used

OpenAI `gpt-6.1-sol` through the Codex harness in T3 Code, with xhigh
reasoning, repository inspection, tool use and code execution. The
context-window size is not exposed by the runtime.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [ ] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-04 11:55:16 -06:00
DottaandPaperclip 4a2c0e5dd9 test(evals): reject unsafe company optional-note placeholders
Keep the safe note label and reject a rendered request for credential text; preserve company oracle v4.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-04 11:24:20 -06:00