Commit Graph
1943 Commits
Author SHA1 Message Date
DottaandPaperclip afcbfe1335 fix: preserve canonical history on managed plugin reset
Record explicit managed-agent instruction resets through the canonical revision CAS path under exact plugin ownership and capability checks. Preserve prior content and plugin audit identity without synthetic user authority.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-09-27 21:38:40 -05:00
DottaandPaperclip 7bed702b59 fix(instructions): refresh canonical content at safe run boundaries
Use the committed database snapshot for native instruction bundles when disk projection is stale. Restage completed stopped working copies from the current head, while preserving live warm owners and original CAS fences for uncollected edits. Cover native stale projection, repeated stopped retries, resolved candidates, and no silent rebasing.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-09-27 21:38:40 -05:00
DottaandPaperclip 2321ddd573 fix(recovery): project exact native run activity across task views
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-09-27 21:38:40 -05:00
DottaandPaperclip 1c4dbec4b0 fix: separate journal proof queue and execution budgets
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-09-27 20:48:50 -05:00
DottaandPaperclip a450a48be4 fix(runtime): await bounded journal proofs with ownership rechecks
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-09-27 20:32:26 -05:00
DottaandPaperclip 8976f3d25c test(runtime): qualify near-limit bounded journal frames
Exercise a 190 MiB journal made of individually bounded event frames and document off-thread proof quotas, retained evidence, and nonblocking file admission.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-09-27 20:32:26 -05:00
DottaandPaperclip cfc3e6805c fix(runtime): retain only journal authority evidence
Project ordinary semantic outputs and unused event headers away while preserving completion, process, and warm-transition authority. Account retained JSON bytes rather than discarded parser work, and open proof inputs without blocking on special files.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-09-27 20:32:26 -05:00
DottaandPaperclip 77ebe7829e test(runtime): reject FIFOs before journal worker deadlines
Exercise both scan and projection through a warmed real worker. Verify a writer-less FIFO is rejected and close completes, with a controlled finally writer release so the pre-fix blocking-open regression cannot strand CI.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-09-27 20:32:26 -05:00
DottaandPaperclip 6f07e711ec fix(runtime): isolate journal proof work in a bounded worker
Use one off-thread proof reader with a bounded queue, monotonic deadlines, heap limits and joined termination. Preserve raw hashes and canonical base64 fingerprints, and prove event-loop responsiveness on a 190 MiB journal.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-09-27 20:32:26 -05:00
DottaandPaperclip 6e525a7b92 fix(runtime): bound durable journal proof memory
Stream proof reads through a fixed buffer, retain only authority evidence, and preserve full-file digests and warm-transition fingerprints. Fail closed when essential evidence exceeds the bounded projection budget.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-09-27 20:04:24 -05:00
DottaandPaperclip e36e59c712 fix: resume exact sandbox lifecycle for export repair
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-09-27 19:54:44 -05:00
DottaandPaperclip 185238b172 fix: repair retained workspace exports without replaying provider work
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-09-27 19:33:16 -05:00
DottaandPaperclip 69549d4248 fix(api): document preserved instruction candidate endpoints
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-09-27 18:55:53 -05:00
DottaandPaperclip 14c3d810c9 test: admit active coordinator in combined export fixture
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-09-27 18:46:03 -05:00
DottaandPaperclip d0d0003d51 fix: recheck native recovery admission under workspace ownership
Preserve terminal unsafe-archive repairs and future retry delays when an earlier reconciler snapshot enters after the live owner releases copyback.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-09-27 18:45:59 -05:00
DottaandPaperclip ffac79eb55 test: seed instruction drift fixture without versioned writer bypass
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-09-27 18:33:40 -05:00
DottaandPaperclip 4340f51e16 fix(instructions): persist registered edits after provider shutdown
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-09-27 18:33:40 -05:00
DottaandPaperclip 3943048383 feat: review and resolve preserved agent instruction edits
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-09-27 18:33:11 -05:00
DottaandPaperclip 5b8dd74c08 feat(runner): expose canonical instruction revision tools
Add native read, update, history, and restore operations with server-bound identity and revision compare-and-swap safety. Register the default catalog and API discovery paths; cover real authority authorization and exact revision restore.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-09-27 18:32:34 -05:00
Fry (Paperclip) 7dfb327bb7 feat(instructions): persist canonical revisions with authorized history and restore
Implement PAP-66 foundation: exact content history, transactional compare-and-swap saves, current responsible-user authorization, recoverable materialization, and editor history. Keep run staging and cleanup integration with PAP-63.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-09-27 18:32:34 -05:00
DottaandPaperclip c2c9996e81 test: match ownership fixture callback and acquired result
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-09-27 18:31:36 -05:00
DottaandPaperclip b727a5659d test: compose unsafe export fixture with finalization ownership
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-09-27 18:28:32 -05:00
DottaandPaperclip 9e3b48fc0c test(runner): persist regression journal through public control-plane API
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-09-27 18:28:32 -05:00
DottaandPaperclip d02e0f45e7 fix: preserve agent-owned native result recovery
Keep settled and foreign-owner failures fenced while allowing audit-only malformed-result outcomes to materialize their existing bounded recovery action.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-09-27 18:26:54 -05:00
DottaandPaperclip c5b2a16a28 fix: recover joined workspace ownership after database reconnect
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-09-27 18:26:54 -05:00
DottaandPaperclip 51f200243f fix: require copyback stop proof after abrupt controller death
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-09-27 18:26:54 -05:00
DottaandPaperclip 2b745708bd fix: serialize native workspace finalization and preserve committed outcomes
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-09-27 18:26:54 -05:00
DottaandPaperclip 372823f0f7 fix(runner): accept valid large journals during session continuation
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-09-27 18:25:16 -05:00
DottaandPaperclip 3564925508 Bound saved API text pages with authenticated storage ranges
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-09-27 18:25:16 -05:00
DottaandPaperclip f1ced64df0 fix(runtime): stop retrying unsafe native workspace archives
Preserve the accepted result and retained sandbox, distinguish repairable unsafe paths from missing sandboxes, and keep transient copy-back failures retryable.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-09-27 16:03:53 -05:00
DottaandPaperclip c2744f77b4 fix(runner): read saved API responses in bounded text pages
Keep large response artifacts inspectable through the existing authenticated API route. Preserve company and run authority, UTF-8 boundaries, and transfer limits. Reproduce the artifact loop through real HTTP and add an explicit full-stack fixture.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-09-27 16:03:15 -05:00
Paperclip 6131a5858f test(workspaces): stream Git output in bootstrap retry fixture
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-09-27 16:00:45 -05:00
Paperclip fab666b131 fix(runtime): preserve restore conflicts and reserve manifest disk space
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-09-27 16:00:45 -05:00
Paperclip 38441fb2e0 fix(runtime): stream workspace snapshots through disk manifests
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-09-27 16:00:45 -05:00
Devin FoleyandPaperclip be43c23e2d fix(recovery): preserve pending native result finalization (#14219)
Preserve native coordinator ownership while accepted results await workspace
copy-back, assessment, or arbitration. Check that ownership in the terminal
update so a coordinator recorded after the liveness read is also protected.
Keep terminal task authority and exhausted-retry cleanup unchanged.

Validation: 28 focused recovery tests, local typecheck/build, 52 passing CI
checks, and Greptile 5/5 with no open findings.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-09-27 00:22:49 -07:00
Devin FoleyandPaperclip 5ee9e751fb fix(runner): discover assigned tools when direct catalogs exceed limits (#14218)
Keep assigned app tools accessible when the combined Runner catalog exceeds
its operation or byte limits. Reserve task and completion tools, then expose
bounded discovery and call tools for large catalogs. Fetch oversized schemas
in reauthorized chunks without blocking later search results.

Retain task ownership, work-mode restrictions, pinned assignments, current
gateway authorization, approvals, and audit. Small catalogs stay direct.

Validation: 46 focused server tests, two Runner capacity tests, local
typecheck/build, 52 passing CI checks, and Greptile 5/5 with no open findings.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-09-27 00:22:16 -07:00
DottaandPaperclip 640dee1802 fix(runner): acquire reusable leases when switching to warm sessions (#14187)
## Thinking Path
> - Paperclip coordinates agent work across task turns.
> - Warm runners need reusable sandbox leases.
> - Lease acquisition read only the environment reuse setting.
> - Startup then read the agent's new warm setting and rejected the
ephemeral lease.
> - This fix requests reuse before acquisition, with provider checks
intact.

## Linked Issues or Issue Description
**What happened?**
Switching an existing native runner task from per-turn to warm fails
with `runner_warm_environment_requires_reusable_lease`. Environment
`reuseLease` defaults to false. Acquisition therefore creates an
ephemeral lease, and capability narrowing correctly denies reuse.

**Expected behavior**
Changing lifecycle between turns requests a compatible lease without
replacing the task workspace or changing the shared environment.

**Steps to reproduce**
1. Start a native runner task with inherited lifecycle and environment
reuse disabled.
2. Change the agent from per-turn to warm.
3. Continue the task on a reusable-capable sandbox provider.

**Paperclip version or commit**
Reproduced on `a6c4e7a`. Related: #12904 established warm workspace
continuity; this fixes the earlier lease-selection mismatch.

## What Changed
- Pass agent settings into acquisition and derive a run-scoped reuse
request.
- Respect explicit environment lifecycle overrides; leave other adapters
unchanged.
- Keep provider capability, ownership, cleanup and restore gates intact.
- Add orchestration and database-backed transition tests; document the
contract.

## Verification
- Red: three new assertions failed before the fix.
- Green: 156 tests pass in environment-run-orchestrator,
environment-runtime, native-sandbox-lifecycle, and
environment-execution-target-capabilities.
- Database-backed regression proves ephemeral → reusable → resumed
lease, same workspace, and unchanged stored environment. Provider RPCs
are mocked, not live Daytona.
- `git diff --check` passes.
- Latest-head CI passes typecheck, build, tests, runner checks, E2E, and
canary dry run. Greptile: 5/5; no open threads.
- Recovery uses the persisted lifecycle. Direct regression: red before,
six cases green after. CI runs the added Vitest cases.
- Full local checks were unavailable (missing dependencies and earlier
memory limits).

## Risks
Warm mode now requests retained resources despite an environment's
default `reuseLease:false`. Existing idle timeout and cleanup still
apply. Unsupported providers remain denied. No migration, credential
[REDACTED], or shared-environment mutation.

## Model Used
OpenAI Codex agent; reasoning, code editing and test execution. Exact
model ID and context size were not exposed to this run.

## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details available)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have described the issue in-PR following the relevant issue
template
- [x] I have not referenced internal/instance-local Paperclip issues or
links
- [x] My branch name describes the change and contains no internal
Paperclip ticket [REDACTED] or instance-derived details
- [x] I have run targeted tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-26 21:23:33 -05:00
DottaandPaperclip f2ed0b65c4 fix(runner): enable API tools by default (#14186)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The native runner exposes tools for company tasks.
> - API search and call tools cover operations without a dedicated tool.
> - The current default hides these tools unless an operator sets an
environment variable.
> - This pull request enables the tools when that variable is absent.
> - Operators can still disable the tools or restrict them to selected
companies.

## Linked Issues or Issue Description

Refs #13003, which added the guarded API tools.

**What happened?**
The native runner does not advertise `search_api` or `call_api` with the
default server configuration.

**Expected behavior**
The tools are available without a special environment variable. Existing
authorization checks still apply.

**Steps to reproduce**
Remove `PAPERCLIP_RUNNER_API_TOOLS_ENABLED` and
`PAPERCLIP_RUNNER_API_TOOLS_COMPANY_IDS`. Create a normal runner
authority. Inspect its tool definitions.

**Paperclip version or commit**
a6c4e7a.

## What Changed

- Enable API tools when the server flag is absent.
- Keep explicit disable, invalid-value rejection, company restrictions,
and binding restrictions.
- Test default tool definitions and run HTTP integration tests without
the enabling flag.
- Update operator and hiring documentation. The existing shared gate
also controls `hire_agent`.
- Use the same policy in the E2E evidence summary so an unset flag is
not reported as disabled.
- Isolate default-availability tests from operator environment
variables.

## Verification

- Red test: two new rollout policy assertions failed before the fix.
- Focused policy, authority, and HTTP tests: 3 files passed; 41 tests
passed, 2 skipped. The two runnerd transport cases require a Rust-built
binary absent from this workspace.
- Command: `pnpm exec vitest run
server/src/services/native-runtime/runner-api-rollout.test.ts
server/src/services/native-runtime/paperclip-runner-tool-authority.test.ts
server/src/services/native-runtime/runner-api.integration.test.ts`.
- Attempted `pnpm -r typecheck`: blocked by missing `cargo` in this
workspace.
- Attempted `pnpm build`: terminated at the 4 GiB memory limit.
- Attempted `pnpm test:run`: stopped after memory pressure to run
focused tests alone.
- Repeated the 41 passing focused tests with an inherited disabled flag
and a foreign-company restriction; test isolation passed.
- `pnpm test:e2e:runner:unit`: 44 files and 543 tests passed.
- `pnpm test:e2e:runner:typecheck` exceeded the workspace memory limit,
including a retry with bounded Go memory settings.
- CI results will be recorded before handoff.

## Risks

- More native runs can discover API tools and the existing `hire_agent`
tool by default.
- The change does not remove company, run, mode, credential, lifecycle,
or approval checks.
- Explicit operator restrictions still take precedence. No database
migration is required.

## Model Used

OpenAI Codex agent. The runtime does not expose the exact model ID or
context-window size. Used reasoning, repository tools, shell execution,
and tests.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-26 21:16:33 -05:00
a6c4e7a8d1 fix(heartbeat): cancel obsolete execution continuations before dispatch (#13761)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A comment can queue execution before the task changes state.
> - The task may be completed, cancelled, deleted, or reassigned before
setup checks ownership.
> - The ownership guard must prevent that obsolete execution from
starting.
> - This pull request records that guard outcome as cancellation and
settles the wake request.
> - Missing history and authorization errors remain failures that
operators can investigate.

## Linked Issues or Issue Description

**What happened?**

A comment can queue a run just before a user marks its task Done. The
ownership guard stops setup before adapter dispatch, but the run becomes
a setup failure and can put the agent into an error state.

**Expected behavior**

Cancel obsolete work when the typed task ownership guard rejects it.
Keep genuine setup errors visible. Do not change the task's terminal
state or current owner.

**Steps to reproduce**

Queue a task continuation, then mark the task Done or Cancelled before
continuation setup. The run should settle as cancelled without invoking
the adapter. Deleting or reassigning the task must also prevent
dispatch. Missing source history or explicit user authorization must
still fail.

**Paperclip version or commit**

Refreshed against master `b2e9e82f053af14777fd68e8834fff6bb64c84c4`.
Related: #13546 covers lost issue-lock claims, and #13888 covers
persisted continuation decisions and retry budgets. This change handles
the final continuation ownership guard during setup.

Thanks to @MrBlackTongue for the original typed cancellation
implementation and regression coverage. This update preserves the
contributor's commits, resolves the master conflict, and narrows
cancellation to task ownership invalidation.

## What Changed

- Add a typed `StaleExecutionContinuationError` for
`continuation_task_ownership_changed`.
- Use existing cancellation settlement for that typed guard, including
the run, wake request, issue execution ownership, and agent state.
Suppress immediate recovery of obsolete work.
- Preserve failure classification for missing source context, missing
user authorization, and untyped errors, including an untyped error with
identical text.
- Cover Done and Cancelled tasks with real database checks. Retain
company and assignment guard coverage. Verify cancellation performs no
adapter dispatch or automatic replay.
- Document the cancellation event and its error classification in the
run-log guide.

## Verification

Current head: `b875486e81`.

- `pnpm -r typecheck`: passed.
- `pnpm build`: passed.
- `pnpm exec vitest run
server/src/__tests__/heartbeat-process-recovery.test.ts
server/src/services/execution-continuation.test.ts --maxWorkers=1
--no-file-parallelism -t 'authorized continuation context|obsolete
continuation setup|untyped continuation setup failures'`: 17 passed; 316
unrelated tests excluded by the name filter.
- `pnpm test:run`: local full run is still finishing. It is not a clean
pass: the observed failures are three company-skills cache checks
(`EACCES` while renaming read-only cache directories on macOS), one
tool-access assertion, and one comment-wake timeout. The three cache
failures reproduce in isolation in unchanged code. The tool-access check
passes in isolation, and the entire comment-wake file passes (29 tests),
including explicit feedback after completion. Final full-run totals will
be added when available.
-
[CI](https://github.com/paperclipai/paperclip/actions/runs/36282532652):
all gates green on this head. An unchanged Runner workspace-diff test
initially returned no diff; an isolated check passed, and the failed job
plus its dependent gate passed on retry with no code change. Original
failed job: 108517010989.
- Greptile: 5/5 on the full current head, with no unresolved review
threads. Branch is current with master and conflict-free.
- Diff check and local secret/PII scan passed. No customer data or
private deployment identifiers are included.

## Risks

The classification change is limited to the typed ownership guard. A
task with missing history remains a failure rather than being treated as
an expected cancellation. Existing company, task-owner, terminal-state,
and authorization checks still prevent dispatch. Cancellation retains
its specific reason in the run log and does not grant a retry. No
schema, dependency, or API changes; no migration is required.

## Model Used

OpenAI GPT-6 via Codex assisted investigation, implementation, code
review, and test execution. The exact serving model identifier and
context-window size are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have described the issue in-PR following the bug issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links
- [x] My branch name describes the change and contains no internal
ticket id
- [x] I have run the focused regression tests locally and they pass;
full-suite results are tracked above
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation
- [x] I have considered and documented the risks
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Devin Foley <devin@paperclip.ing>
2026-09-26 17:43:33 -07:00
Devin FoleyandPaperclip b2e9e82f05 fix: stop remote Grok runs before continuing queued messages (#14100)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The execution service owns each run and saves messages sent while it
runs.
> - Interrupt must stop the current executor before it delivers those
messages.
> - Remote Grok commands did not register the host cancellation control.
> - A cancelled task run could still write Done and prevent queue
recovery.
> - This pull request connects remote cancellation and revokes cancelled
run writes.
> - Saved input can use the existing queue admission rules after
verified cleanup.

## Linked Issues or Issue Description

**What happened?**

Interrupting a queued message marked a remote Grok run cancelled before
its sandbox stopped. The old run could still post a reply and mark the
task Done. Its saved follow-up remained deferred behind execution
recovery.

**Expected behavior**

Stop revokes run write authority and waits for verified termination.
Saved messages remain durable and enter one successor through normal
admission after cleanup.

**Steps to reproduce**

1. Run a task with `grok_local` in a remote sandbox.
2. Send a follow-up and use Interrupt while the command runs.
3. Let the old command attempt a task status update after cancellation.
4. Observe the task disposition and the saved message queue.

**Paperclip version or commit**

The gap is present in master at `d3e0f0a238`.

**Deployment mode**

Authenticated server with a Daytona sandbox.

Related work: #14028 and #14046 handle bounded continuation. #13291
covers infrastructure interruption and verified remote cleanup. #13332
addresses atomic recovery holds. This change handles direct Grok
operator cancellation and stale task writes.

## What Changed

- Register remote Grok cancellation before preparation. Keep command
ownership until the host confirms sandbox termination.
- Reuse the sandbox cancellation boundary for the direct CLI invocation.
Reject fresh attempts after cancellation and preserve workspace restore
failure evidence.
- Reject writes from cancelled task JWTs and runs with a pending stop.
Preserve diagnostic reads and existing conversation error codes.
- Recheck run authority under a database lock before task updates and
interaction responses commit.
- Preserve authorized handoffs that stop their own run. Only the
server-issued stop receipt for that request permits the final task
update.
- Add tests for hung commands, unverified stops, early cancellation,
copy-back failures, late Done, late interaction responses, authorized
handoffs, exact lease receipts, and one queue successor across
concurrent restart sweeps.
- Document the cancellation and write-authority contract.

## Verification

- Targeted adapter, cancellation-boundary, authentication,
queued-message, interaction-service, and activity-route tests passed.
The expanded run passed 214 tests; one new test had an incomplete
fixture. After correcting the fixture, all 8 selected follow-up cases
passed.
- `pnpm -r typecheck`: passed on
`179c86caf1bf0d89914a503d46e24af7e4b8c557`.
- `pnpm build`: passed on the same commit.
- `pnpm test:run`: the general-server group completed with 13,521
passed, 99 skipped, and 18 failed tests. It then stopped, so the
remaining local groups did not run. Five Slack, email, and wake-batching
failures passed on focused reruns after correcting the local
environment. The remaining 13 failures reproduce as `EACCES` on rename
in unchanged skill-cache code on macOS. Two custom-image suite setup
hooks also failed to start embedded PostgreSQL after the machine
exhausted shared-memory slots; all 31 tests in that file passed on rerun
after the local resource issue was resolved. CI covers all test groups.
- CI: 53 checks passed and 2 were skipped on the latest commit,
including the aggregate verification gate. The last server shard passed
on its single rerun after a preview-server startup timeout. The affected
file also passed locally with 28 passed and 3 skipped.
- Greptile: 5/5 on the latest commit. Both review threads are resolved.
- No live deployment or staging task mutation has been performed.

## Risks

- Stopping the sandbox can prevent file copy-back. The result preserves
workspace restore failure evidence; termination does not imply restored
files.
- If provider termination fails, the adapter keeps ownership of its
outstanding command and does not acknowledge Stop.
- The write restriction now applies to ordinary cancelled tasks. Reads
remain allowed. Task and interaction checks add a shared run-row lock to
agent mutations. An exact server-issued receipt permits the task request
that stopped its own run to complete its handoff.
- Existing terminal tasks are not reopened automatically. An operator
must correct a historical late Done before its saved queue can continue.
- No schema migration or UI change.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository inspection, code
execution, and test tools. The precise backend revision and
context-window size are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` OR (b) described the issue in-PR following the relevant issue
template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-26 17:07:07 -07:00
DottaandCodie 01d9a12185 fix: make keyboard shortcut enablement a personal preference (#14141)
Store keyboard shortcut enablement per user and expose it in Profile settings.

Co-Authored-By: Codie <Codie@users.noreply.github.com>
2026-09-26 11:50:04 -05:00
Devin FoleyandPaperclip d3e0f0a238 fix(server): validate CLI auth challenge IDs (#14095)
Reject malformed CLI challenge UUIDs before database access while preserving secret and authentication precedence. Document the HTTP responses and pin the supervisor test fixture to the CI-selected Node executable.

Validated by 21 focused tests, root typecheck/build, and full CI. Mac broad-suite baseline limitations are documented in the PR. Greptile 5/5.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-09-26 00:48:14 -07:00
Devin FoleyandPaperclip 1e3a148f46 fix(connections): retain safe broker rejection diagnostics (#14098)
Preserve fixed broker rejection reason codes within strict size/time limits while retaining public error codes and status. Unknown bodies remain generic; no raw response or credential material enters the error.

Validated by 33 consumer tests, a synthetic producer HTTP contract fixture, root typecheck/build, and full CI. Greptile 5/5.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-09-26 00:47:47 -07:00
Devin FoleyandPaperclip ffa32373bc fix(runtime): honor provider acquisition timeout defaults (#14097)
Declare provider acquisition budgets so slow Daytona creation does not hit the host’s 30-second fallback. Bound creation and setup to one deadline and preserve scoped cleanup ownership after timeout. Legacy drivers retain their original call shape.

Validated by 238 provider/manifest tests, focused database and heartbeat regressions, root and standalone provider typecheck/build, and full CI. Local broad tests also expose recorded Mac baseline limitations. Greptile 5/5.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-09-26 00:47:21 -07:00
Devin FoleyandPaperclip c3ddb288b1 fix: validate work product execution workspace references (#14063)
Reject invalid and cross-company execution workspace references with a useful
422 before changing the work product. Hold the validated reference through
the transaction so concurrent deletion cannot turn validation into a 500.

Verified 13 focused tests, full typecheck/build, and green PR CI. Greptile
5/5 with no unresolved comments. The known UI copy-toast flake passed on an
unchanged-commit retry and in a focused local run.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-09-25 17:06:30 -07:00
DottaandPaperclip 96bf004a79 fix: use persisted state for lifecycle continuation and retry budgets (#13888)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Its control plane decides when a task can continue, wait, stop, or
complete.
> - Legacy continuation could change when an agent changed its wording
without changing task state.
> - Shared attempt counts also let repair and infrastructure retries
affect each other's limits.
> - This pull request uses persisted state and separate, bounded
allowances for these decisions.
> - If automatic repair stops, the task explains what happened and
offers a guarded retry.
> - Paired tests and real-provider evaluations verify that Stop,
approvals, ownership, and spending limits remain authoritative.

## Linked Issues or Issue Description

Related work: Refs #13761, Refs #11126, Refs #13610. These cover
obsolete continuation dispatch and retry storms. Open and closed issues
and PRs were searched for related lifecycle, continuation, and retry
work.

**What happened?**
Legacy continuation depended on English wording and progress heuristics.
Repair, failure retry, and productive continuation could consume shared
counts. When bounded repair stopped, the task showed a technical
recovery message without a clear next action.

**Expected behavior**
Persisted disposition and owned execution paths determine the next
action. Missing disposition prompts bounded agent repair. Explicit work
mode determines planning mode. Narrative changes and raw activity counts
cannot replenish allowances. An exhausted repair shows a readable
notice. An explicit retry checks current controls and preserves the
assigned agent.

**Steps to reproduce**
Run `pnpm test:lifecycle-baseline`. The paired probes keep structured
state constant while varying completion, planning, blocker, and progress
prose. Run the explicit `lifecycle-baseline` and
`continuation-accounting` Product E2E suites for real-provider coverage.
In Storybook, open **Design previews / Recovery notice** to inspect the
production component's normal, pending, acknowledged, unavailable,
failure, and mobile states.

## What Changed

- Hide the image attachment button, icon, and drop/paste hint in answer
composers. Image paste and drop support remains available.
- Merge current master and retain both browser regression sets. Use a
production-stamped service worker in the offline recovery browser
fixture.
- Share one state-based legacy continuation decision across immediate,
delayed, and recovered dispatch. Bind bounded repairs to their source
run and episode.
- Remove title and description wording from work-mode authority. Agents
can still write requested plans in execution mode.
- Persist separate failure-retry and productive-continuation counters.
Disposition repair and resource waits cannot consume or reset those
allowances.
- Validate delayed repair identity, then recheck current gates before
provider dispatch. Fence native startup cancellation.
- Show **Agent needs attention**, a plain-language explanation, **Retry
agent**, and expandable details in both task interfaces. Report request
progress, acknowledgement, and errors inline.
- Store typed recovery notice metadata. Recognize older active notices
only through exact stored action and run IDs. Notice text never grants
retry authority.
- Use the existing recovery-action endpoint for retry. Recheck current
action, status, owner, agent availability, dependencies, active runs,
pending questions and confirmations, approvals, pause controls, and
budget. Duplicate requests do not wake twice.
- Add component, page, route, database, contract, and Storybook
coverage. Keep the scenario inventory and executable evals here.
Historical reports and snapshots live in the [commit-pinned
paperclip-evals
archive](https://github.com/paperclipai/paperclip-evals/blob/ce3e5afcd4a1184650f586a2b5b8be5874c66c8b/experiments/2026-09-lifecycle-authority/README.md).
- Preserve unsaved project fields while the same project URL changes to
its canonical alias. Do not reuse data across projects or companies.
This separate fix addresses the repeated repository-editor browser
failure without changing the browser test.
- Keep the development service worker from intercepting Vite module
reloads. Update the connection-intent browser fixture to record progress
and completion through the agent API.

## Verification

Merge preparation on September 25, commit
`c1e8e4b7ddd9fbc4913ed55ce21b8e12906c2f97`:

- Merged master `bd2030932` and resolved the browser test-list conflict
by keeping both sets of regressions.
- Deterministic lifecycle baseline: 1,090/1,090 assertions passed; no
failures, skips, or missing selected evidence. Unit 423, runner 184,
database integration 397, grading 86.
- Browser support: 17/17 passed. The offline recovery test first failed
with an unstamped development worker, then passed with the production
stamp. Its assertions are unchanged.
- Focused interaction UI and offline fallback tests: 19/19 passed.
Verified the custom-answer composer in Storybook: no attachment controls
or hint; entering an answer enables Next.
- Recursive typecheck, production build, token gates, and diff checks
passed. The worktree is clean. No new real-provider campaign was run.
- Current CI and review: [Current PR CI
passed](https://github.com/paperclipai/paperclip/actions/runs/36166011243):
55 successful checks and two optional Storybook skips. Greptile scored
this exact commit 5/5. Hiding the question attachment controls is an
intentional UI change; paste/drop remains available.

Earlier recovery UI verification, commit
`21be0fec0e90e86b6d662b8ee4831847cd041cdb`:

- Recursive typecheck, production build, token gates, and diff checks
passed.
- Focused UI coverage: 338 tests passed across six suites (336 before
the interaction guard, with the two affected suites rerun at 149 passed
after it). Covers both task interfaces, the real page mutation,
pending/error acknowledgement, stale state, and unavailable controls.
- Recovery database integration: 352 tests passed before the interaction
guard. The complete recovery-action and mutation-route suites passed 181
tests after it. The two new pending question/confirmation regressions
failed before the fix and passed afterward, including
resolved-interaction controls. Shared validator suite: 31 passed. E2E
catalog suites: 34 passed.
- Browser inspection passed for light/dark themes, mobile layout,
expandable details, pending retry, acknowledgement, failure, and
disabled retry. Storybook renders the production component; its request
is simulated.
- The broad local run hit two chat callback-order wait failures and was
stopped after all CI unit/database/runner shards passed. Both local
failures passed when rerun without the competing full-suite process.
- CI exposed a repeated project-repository draft-loss race during
canonical redirects. A new unit regression failed before the fix; all
nine project-page tests now pass, including controls for other projects
and companies. Both unchanged repository browser tests passed against a
fresh local server. UI typecheck, production UI build, and token gates
passed after this fix.
- [Earlier PR CI
passed](https://github.com/paperclipai/paperclip/actions/runs/36072486798)
on `21be0fec0e90e86b6d662b8ee4831847cd041cdb`: 55 successful checks, two
optional Storybook skips, and no failed or pending checks. The
repository browser shard passed with the production fix. Greptile is 5/5
on this exact commit with no unresolved review threads. The PR is
mergeable.

Historical, source-qualified lifecycle evidence:

- Lifecycle baseline: 1,074 assertions. Native session coverage: 447
tests. Product E2E support: 515 tests. Browser support: 11 tests. Full
earlier verification is retained in the archive.
- [Real-provider campaign: 8/8 passed, zero
retries](https://d1p6rlowie26tp.cloudfront.net/runner-e2e/campaigns/gha-35881382080-1/index.html),
source `e88d210417280140b44a36449027290adcb1aeaa`. Evidence and cleanup
checks passed. This includes deliberately exhausted repair cases that
correctly remain blocked; it does not mean every task finished Done.
This campaign predates the recovery UI change.
- Archive migration verified all 16 original JSON files byte-for-byte
and all 24 checksum entries. App tests do not need private archive
access. [Archive PR
#27](https://github.com/paperclipai/paperclip-evals/pull/27) is merged.

## Risks

- Agents that omit durable disposition receive at most two repair
attempts by default. Prose-only completion exposes missing state rather
than silently changing scheduling.
- A retry is an explicit board action. The server rechecks current
controls. A successful response confirms the task returned to To do; it
does not claim that the provider has already started.
- Existing notice metadata remains valid. Only older active notices with
matching structured evidence receive the new UI. Historical notices
without that evidence keep their existing rendering. No schema migration
is required.
- Old run records require conservative retry accounting. Tests cover old
counters, alternating retry lanes, restarts, and exhausted repairs.
- Historical snapshots require private `paperclip-evals` access. The app
index retains public campaign links. Live campaigns qualify specific
sources and scenarios; no new real-provider campaign has run for the
recovery UI commit.

> This fixes existing lifecycle and recovery behavior and does not
duplicate planned core work.

## Model Used

OpenAI GPT-6 through Codex assisted implementation, reasoning, code
execution, and review. The exact serving model ID and context window are
not exposed in this task. Historical real-provider evaluations used
Codex model `gpt-5.6-sol`, separately from the implementation assistant.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-25 15:28:11 -07:00
Devin FoleyandPaperclip 6bc830b62b fix(recovery): escalate an issue whose interrupted run has spent its retry budget (#14046)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The stranded-issue sweeper is what puts an assigned issue back on a
live path when its run dies.
> - A failed or interrupted run gets a bounded transient retry; when
that budget is spent, the retry scheduler queues nothing and reports the
exhaustion.
> - The sweeper treated that "nothing queued" like every other "nothing
queued" and skipped the issue, on every tick, forever: `in_progress`, no
run, no path, no notice.
> - Three server restarts in a row (a deploy storm) are enough to spend
the budget, so this is reachable in ordinary operation.
> - This pull request makes the sweeper escalate a spent budget to a
board-owned recovery action, the same visible `blocked` state its other
dead ends use.
> - The benefit is that no issue can sit assigned and silent after its
retries run out.

## Linked Issues or Issue Description

No existing issue found. Related work: #14028 (this branch's
predecessor: queue drain-time wakes, retry interrupted corrective runs)
fixed two neighbouring gaps but not this one.

**What happened?**

A routine-created issue's run was interrupted by a graceful server
shutdown, and both bounded transient retries were interrupted by the
next two shutdowns. The retry scheduler logged "Bounded retry exhausted
after 2 scheduled attempts; no further automatic retry will be queued"
and stopped. On every tick after that, `reconcileStrandedAssignedIssues`
reached the generic continuation lane, `enqueueStrandedIssueRecovery`
delegated to `scheduleRecoveryRetry`, which returned null (exhausted),
and the sweeper counted the issue as `skipped`. The issue stayed
`in_progress` with no run, no recovery action, and no comment for hours
until a person woke the agent by hand. The same hole exists in the
assigned-`todo` dispatch lane.

**Expected behavior**

When the transient retry budget for an interrupted or failed run is
spent, the sweeper should treat it as the dead end it is: escalate the
issue to `blocked` with a board-owned recovery action and a notice,
exactly as it does when a continuation retry chain or an assignment
retry chain is exhausted. Leaving the budget spent across restarts is
correct and unchanged; leaving the issue silent is not.

**Steps to reproduce**

1. Assign an agent using a conversation adapter (its interrupted runs
carry `conversationContinuation`, so legacy reconciliation does not
terminalize them) an issue and let a run start.
2. Interrupt the run with a graceful shutdown, let the transient retry
start, interrupt it, and repeat once more so `scheduledRetryAttempt`
reaches 2.
3. Run the stranded-issue sweep. Before this change: `skipped` every
tick, issue `in_progress`, no run, no recovery action. After:
`escalated`, issue `blocked`, one active board-owned
`issue_recovery_actions` row, a "No live execution path" notice.

**Paperclip version or commit**

master at bd6caf51bb (2026-09-25).

**Deployment mode**

Managed cloud instance restarted by fleet deploys; the code path is the
same for any operator whose server restarts more often than the retry
budget allows.

## What Changed

- `recoveryService` gains an optional `transientRetryBudgetSpent(run)`
dependency; `heartbeatService` wires it as
`executionFailureRetryCount(run) >=
BOUNDED_TRANSIENT_HEARTBEAT_RETRY_MAX_ATTEMPTS`, the same check
`scheduleBoundedRetryForRun` applies.
- `enqueueStrandedIssueRecovery` takes an optional `outcome`
out-parameter and sets `retryExhausted` when the failed predecessor's
retry returned nothing **because** the budget is spent. A null return
without it still means another authority owns the run (native runtime,
legacy reconciliation) and the caller leaves it alone; the deliberate
"failure recovery cannot fall through into the continuation queue" rule
is unchanged.
- The generic `in_progress` continuation lane and the assigned-`todo`
dispatch lane escalate on `retryExhausted` via
`escalateStrandedAssignedIssue` with a "No live execution path" notice
(danger tone), which creates the board-owned source-scoped recovery
action and moves the issue to `blocked`. Every other
`enqueueStrandedIssueRecovery` caller is unchanged.
- Tests (`heartbeat-process-recovery.test.ts`): an `in_progress` issue
with a spent budget escalates (blocked, one active board-owned action,
notice, no successor run, idempotent on the next sweep); an interrupted
run with budget remaining still gets its transient retry; an assigned
`todo` issue with a spent dispatch budget escalates. The existing guard
"does not reset an exhausted incident budget on server restart" keeps
its no-successor-run assertion and now expects the board escalation
instead of nothing, with a comment on why.

## Verification

```
pnpm -r --filter './packages/**' build
cd server
npx vitest run src/__tests__/heartbeat-process-recovery.test.ts \
  src/__tests__/issue-recovery-actions.test.ts \
  src/services/recovery/successful-run-handoff.test.ts \
  src/__tests__/heartbeat-task-drain-admission-release.test.ts \
  src/__tests__/attention-service.test.ts \
  src/__tests__/heartbeat-comment-wake-batching.test.ts
npx tsc --noEmit -p tsconfig.json
```

Live reproduction: the exact stuck state (issue `in_progress`, latest
run `interrupted` with `scheduledRetryAttempt` 2, the "Bounded retry
exhausted" lifecycle event, sweeper `skipped` every tick) was observed
on a managed instance running current master before this change was
written.

## Risks

- Behavior change is limited to runs whose transient budget is already
spent, which previously produced no action at all. Nothing new is
retried; the change only adds the escalation, so no retry loop can be
introduced.
- The board-owned action spawns no run. Resolving it (restore to the
owner) re-dispatches through the existing recovery-action routes, the
same flow as every other stranded escalation.
- Native-runtime and legacy-reconciliation predecessors are untouched:
they return before the exhaustion check.

## Model Used

Claude (Anthropic) — `claude-fable-5-1`, extended thinking, tool use
(Claude Code CLI). The incident diagnosis and the choice to escalate
rather than re-dispatch were steered by the maintainer.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-25 13:17:48 -07:00
Devin FoleyandPaperclip bd6caf51bb fix: preserve restore failure results and stop unsafe retries (#14035)
Preserve agent output and earlier execution errors when workspace restore fails. Report the restore phase and confirmed saved-plan links. Require verified repair before retrying unsafe archives, while preserving approval states and the retry budget.

Verified with full CI, 506 focused regression tests, and Greptile 5/5 with all review threads resolved.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-09-25 11:34:31 -07:00
Devin FoleyandPaperclip 5b09d66183 fix(heartbeat): keep drain-time wakes queued and retry interrupted disposition handoffs (#14028)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The server's heartbeat scheduler queues, admits, and recovers agent
runs.
> - A control plane arms an instance task drain before it restarts the
process, so no new run starts mid-restart.
> - Two recovery paths treated that transient hold as a permanent
verdict: a wake that arrived during a drain was written as `skipped` and
never replayed, and a corrective disposition run that the restart
interrupted counted as an exhausted attempt, so the issue went to
`blocked`.
> - Both outcomes leave an assigned issue with no run and no path until
a person notices.
> - This pull request keeps drain-time wakes in the durable queue and
gives an interrupted corrective run the same bounded transient retry any
interrupted run gets.
> - The benefit is that a graceful restart never strands or blocks an
issue by itself.

## Linked Issues or Issue Description

No existing issue found. I searched open PRs that touch the task drain
(#13527 adds opt-in termination of active runs; #13965 handles deferred
wakes after a stale cancellation). Neither replays drain-time wakes or
changes the corrective-handoff escalation.

**What happened?**

1. An operator accepted a plan confirmation while the instance was in a
task drain (a control plane armed it before a deploy restart).
`enqueueWakeup` wrote the assignee's continuation wake with `status:
"skipped"` and `heartbeatSkip.reason: "task_drain"`. The drain is
process-local and cleared on the restart seconds later, but nothing
replays skipped wakes. The issue stayed in `todo` with no run for 20
hours until a person retried it by hand. The stranded-issue sweeper did
not help: a `todo` issue whose latest run succeeded is treated as
deliberately handed back.
2. On another issue, the watchdog correctly raised
`successful_run_missing_state` and queued the single corrective handoff
run. A graceful server shutdown (the same deploy restart) interrupted
that run with `server_shutdown_interrupted`. On the next boot,
`reconcileStrandedAssignedIssues` saw a corrective run at
`handoffAttempt >= maxHandoffAttempts` and escalated the issue to
`blocked` with a board-owned `missing_disposition` recovery action. The
agent never got to finish one corrective attempt.

**Expected behavior**

A task drain holds admission, not the request. A wake that arrives
during a drain must run once the drain lifts or the process restarts. An
interrupted corrective run is not evidence that the agent could not
choose a disposition; it should be retried like any interrupted run, and
escalate only when that retry budget is spent or a finished attempt
still leaves no disposition.

**Steps to reproduce**

1. Assign an agent an issue and `POST /api/instance/task-drain` with a
Cloud control assertion (or call `startTaskDrain({})` in a test).
2. Trigger any wake for that agent (assignment, comment, or an accepted
`request_confirmation`).
3. Observe the `agent_wakeup_requests` row: `status = skipped`, `reason
= heartbeat.scheduling_suppressed`, `payload.heartbeatSkip.reason =
task_drain`. Stop the drain or restart: no run is ever created for that
wake.
4. For the second case: let a `finish_successful_run_handoff` run be
interrupted by SIGTERM during a graceful shutdown, then boot. The issue
moves to `blocked` on the startup sweep without any retry.

**Paperclip version or commit**

master at e2f1a66aa7 (2026-09-25).

**Deployment mode**

Managed cloud instance behind a control plane that arms task drains
before deploy restarts; the code paths are the same for any operator who
uses the drain endpoint.

## What Changed

- `enqueueWakeup` no longer writes a wake as `skipped` when the only
suppression is `task_drain`. The wake and its queued run land in the
durable queue; `startNextQueuedRunForAgent` and `executeRun` already
refuse to admit work while the drain is active, and the boot-time
`resumeQueuedRuns` pass picks it up after the restart.
`worktree_instance` and `database_restore_in_progress` suppression still
write `skipped`: those holds are not transient restarts.
- `reconcileStrandedAssignedIssues`: when the latest run is a corrective
successful-run handoff at its attempt cap **and** that run is
`interrupted`, the sweeper schedules the bounded transient retry (via
`enqueueStrandedIssueRecovery` → `scheduleRecoveryRetry`) instead of
escalating. The retry keeps the handoff context, so it is still the
corrective run. If the retry budget is spent, or a finished attempt
still leaves no disposition, escalation proceeds exactly as before.
Failed corrective runs (adapter or provider failures) are unchanged:
they still escalate immediately.
- New sweep counter `successfulRunHandoffRetried`, included in the
startup and periodic recovery log lines.
- Tests: `heartbeat-task-drain-admission-release.test.ts` gains a case
that a wake during a drain is queued, held while the drain is active
(quiescence still reports true), and runs to completion once the drain
lifts. `heartbeat-process-recovery.test.ts` gains two cases: an
interrupted corrective run is retried with its handoff context and the
issue stays `in_progress`; an interrupted corrective run with a spent
transient budget escalates to `blocked` with the usual recovery action
evidence.

## Verification

```
pnpm -r --filter './packages/**' build
cd server
npx vitest run src/__tests__/heartbeat-process-recovery.test.ts            # 296 passed
npx vitest run src/__tests__/heartbeat-task-drain-admission-release.test.ts \
  src/__tests__/heartbeat-worktree-suppression.test.ts \
  src/__tests__/heartbeat-scheduling-suppression.test.ts \
  src/__tests__/heartbeat-task-drain.test.ts \
  src/__tests__/instance-settings-routes.test.ts \
  src/services/recovery/successful-run-handoff.test.ts \
  src/__tests__/issue-recovery-actions.test.ts \
  src/__tests__/heartbeat-comment-wake-batching.test.ts \
  src/__tests__/attention-service.test.ts                                    # 226 passed
npx tsc --noEmit -p tsconfig.json                                           # clean
```

Live reproduction of the first defect: after this change was written, a
board wake issued against a draining managed instance (running current
master) was again recorded as `skipped` with `heartbeatSkip.reason =
task_drain`, which is the exact row the new test asserts no longer
appears.

## Risks

- A wake that lands during a drain now waits in the queue instead of
being dropped. `getTaskDrainStatus().pendingWakes` counts only in-flight
enqueue promises, so quiescence is unchanged and a control plane waiting
for the drain is not held longer. If a drain is stopped without a
restart, the queued run starts on the next scheduler tick.
- The interrupted-handoff retry reuses the existing transient retry
budget (two attempts) and the existing `hasActiveExecutionPath` skip, so
a sweep cannot double-schedule. Native-runtime corrective runs still
return `null` from the recovery enqueue and escalate as before.
- Self-hosted instances that never call the drain endpoint see no change
on the first path; the second path only changes behavior for corrective
runs interrupted by a graceful shutdown.

## Model Used

Claude (Anthropic) — `claude-fable-5-1`, extended thinking, tool use
(Claude Code CLI). Human-directed: diagnosis of the two production
incidents, the fix design, and review were steered by the maintainer.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-25 11:28:54 -07:00
Devin FoleyandPaperclip e2f1a66aa7 feat: support default-hidden experimental settings (#13980)
## Thinking Path

> - Paperclip is the open source control plane for AI-agent companies.
> - Operators can hide settings that their users must not change.
> - The server shares the effective restrictions with the UI and
settings API.
> - An explicit list must change whenever a new experimental flag is
added.
> - This pull request adds a wildcard with named exceptions to the
existing setting.
> - Core applies the policy to its current catalog, so new flags stay
hidden automatically.

## Linked Issues or Issue Description

Related: #11823 introduced settings visibility. #13907 added workspace
isolation visibility. I searched related PRs and issues and found no
duplicate wildcard implementation.

**What existing behavior does this improve?**

Operator control of experimental setting visibility through
`PAPERCLIP_HIDDEN_SETTINGS`.

**Subsystem affected**

Shared settings policy and its existing server health and mutation
consumers.

**Current behavior**

Operators must name every hidden experimental toggle. A new Core flag
can become visible until the operator updates that list.

**Proposed behavior**

`instance.experimental.*` hides current and future experimental toggles.
Entries such as `!instance.experimental.enableEnvironments` leave named
controls available. Explicit hidden keys and the hidden parent page take
precedence over exceptions.

**Reason and benefit**

Operators can maintain a short list of allowed controls instead of a
second copy of Core's full feature catalog.

**Breaking changes**

Existing explicit lists and unset configuration keep their behavior. The
new syntax is opt-in. Older images ignore it, so operators must retain
explicit restrictions until those images are upgraded. Visibility does
not change feature values.

## What Changed

- Expand the wildcard into concrete catalog keys in the shared parser.
- Limit exceptions to known experimental controls and preserve explicit
restrictions in either input order.
- Test a synthetic future catalog addition, duplicate and invalid
entries, API rejection, same-value echoes, and the effective health
payload.
- Document the syntax and the transition for deployments with mixed
image versions.

## Verification

- Targeted parser, future-catalog, health, and settings-route tests
pass: 95 tests across four files.
- `pnpm -r typecheck` passes, including Rust checks, with the installed
Cargo directory on PATH.
- `pnpm build` passes.
- All current-head CI gates pass, including the full test shards, Rust,
build, browser E2E, and canary dry run:
https://github.com/paperclipai/paperclip/actions/runs/36083292578. One
unchanged runtime-exposure cold-start test passed on its first retry.
- The full local `pnpm test:run` did not pass on macOS/Node 25: the
first server group reported 13,356 passed, 18 failed, and 99 skipped,
with six failed files (including two failed suite setups). Failures were
in unchanged runtime/company skill cache, chat/email connector fixtures,
embedded-Postgres setup, and workspace cleanup tests. A standalone
filesystem probe reproduced the read-only-directory rename permission
failure. Missing connector fixture paths, database startup failures, and
two integration assertions also occurred; the remaining local groups
were not reached after this group failed. The corresponding CI lanes all
pass. These local failures are not claimed as fixed by this PR.
- Browser suites were not run because this changes the shared policy,
not UI rendering or browser workflows. Health payload and route tests
cover the shared UI/API contract.

## Risks

A malformed exception remains hidden and is reported as unknown.
Exceptions cannot override an explicit hidden toggle or parent page.
Older images ignore wildcard syntax; keep their explicit list during a
mixed-version rollout. No schema or feature-value changes are included.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, tool use, and code
execution. The exact runtime variant and context-window size are not
exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-25 08:07:50 -07:00