mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-21 10:52:06 +01:00
9bc90cb189bbb0f31206c0baeda6286be052f103
299
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
417802ca87 |
fix(plugins): refresh unchanged-version bundle descriptions (#22295)
Refresh known display descriptions in same-version bundled plugin registrations only when every behavioral field matches. Compare manifests in their persisted JSON form and await bundle reconciliation before final volume-free package qualification. Verified 77 focused tests, server typechecking, and all 59 successful PR checks. Greptile 5/5 with no unresolved review comments. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
|
a7ea2a799a |
fix(ephemeral): save retained runtime skill bundles at startup and never strand a volume-free server on lost ones (#22190)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A volume-free server keeps no local volume. It must find every file it needs in durable storage. > - Before a volume-free server starts, a data gate refuses to start if records still point at files on a local disk. > - Old native runs can keep runtime skill bundles that a release before #19729 wrote only to its own disk. The gate reports these as `runtime_skill_assets_require_migration`. > - On a volume-free server those files are not on the disk, and no volume can be attached again. So the refusal repeats on every restart, and nothing can clear it. A staging stack crash-looped like this and stayed down. > - This pull request makes the server copy every retained bundle that is still on its disk into durable storage, and start over the bundles that are gone, the same way #20990 handles harness backups. > - The benefit is that a volume-free stack can always start again. No one has to repair it by hand. ## Linked Issues or Issue Description Refs #19675, #19729, #20990 **What happened?** A volume-free staging stack ran for twelve days on a release that kept runtime skill bundles only on its container disk. When that container stopped, every newer release crash-looped at boot with `Volume-free server data is not eligible: runtime_skill_assets_require_migration`. The old container was gone, so the bundles were gone. The deploy could not roll back, and a retry could not help. **Expected behavior** A volume-free server starts. It saves each retained bundle that is still on its disk to durable storage. It starts over a bundle that is not on its disk when the volume-free owner marker proves that no volume is mounted, and restart recovery owns those runs. **Steps to reproduce** 1. Retain a native run with an admitted runtime skill bundle that has no version ID and no durable preservation receipt. 2. Remove the bundle from the instance root. 3. Start the server with `PAPERCLIP_EPHEMERAL_SERVER=true` and the volume-free owner marker present. 4. Startup fails with `runtime_skill_assets_require_migration`, on every restart. ## What Changed - `runtime-skill-preservation.ts`: - The obligation scan is one generator. `hasLocalRuntimeSkillObligations` uses it. - New `preserveLocalRuntimeSkillObligations` copies each obligated run's bundles with `preserveRuntimeSkillAssets` and returns `{ bundles, unsaved }`. - A missing source is decided at its reader: `readLocalManifest`/`readSource` throw `RuntimeSkillSourceAbsent` (other instance root, or a missing directory, manifest or listed file). Only that error marks a run unsaved. Any other failure, including an `ENOENT` from storage or cache work, propagates. - `preserveRuntimeSkillAssets` no longer aborts on an invalid or non-durable existing receipt. It copies the local bytes again and replaces the receipt. - One durability helper (`preservedAssetRowDurable` for metadata, `preservedAssetDurable` with a metadata or object read) replaces the copies in the scan, staging, locking, preservation and rehydration. - `runtime-skill-preservation-reference.ts`: the scan matches only null pins, which is exactly what preservation copies. A non-null pin is a version pin and cannot become an obligation that nothing can settle. - `file-manifest-cache.ts`: new `inspectFileManifest` returns `verified`, `missing` or `changed` from the same walk. `verifyFileManifest` wraps it. - `ephemeral-server-contract.ts`: the gate saves the bundles still on the disk. If none stay unsaved, the blocker clears. Unsaved runs are waived only when the volume-free owner marker is present, with a warning. A save failure still refuses. The decision uses the copy pass result, not another scan. One exported blocker constant. - `ephemeral-migration-data.ts`: offline enrollment accepts the blocker and copies the bundles from the attached volume, in a phase after the package phases. Any unsaved run still fails the enrollment. - `index.ts`: comment update only. - Tests: `ephemeral-server-runtime-skill-startup.test.ts` (real scan, real copy, embedded Postgres) and an S3-gated case in `ephemeral-migration.test.ts`. ## Verification - `cd server && npx vitest run src/__tests__/ephemeral-server-runtime-skill-startup.test.ts` - Red on `origin/master`: 2 of 3 fail with the production error `Volume-free server data is not eligible: runtime_skill_assets_require_migration`. - Green with this change: all cases pass (lost bundle, saved bundle, no marker, mixed present and lost, one missing listed file, non-version pin, mixed pins, non-durable receipt recapture, storage failure, storage ENOENT, unsaved set). - `cd server && npx vitest run src/__tests__/ephemeral-server-startup-run-state.test.ts src/__tests__/runtime-skill-preservation.test.ts` pass (existing gate and preservation behavior unchanged). - `cd server && npx tsc --noEmit -p .` passes. - `ephemeral-migration.test.ts` needs `PAPERCLIP_TEST_S3_ENDPOINT`. It typechecks, but it was not run here. ## Risks - A run whose bundle is lost now starts under restart recovery instead of blocking the whole server. Recovery of that one run can fail. Before, the whole server stayed down, and the bundle was lost anyway. - Startup does more work only when unsaved legacy bundles exist. A current release saves bundles before admission (#19729), so new runs never add this work. - No schema change. No API change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
d1b5cfe038 |
feat(connections): let users delete data read through a connection (#22010)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents call connected apps, such as Google Workspace, through
Paperclip's tool gateway
> - Paperclip saves part of each tool result, approval previews, app
views and the full run transcripts that contain tool output
> - No user action deleted this data. Removing the connection left it in
place
> - Google's restricted-scope review requires that users can delete the
data an app obtained, and that the privacy policy says how
> - This pull request adds "Delete data read through this connection"
for any connection, on the connection's page and as an option when you
remove it
> - The benefit is that the picker text ("They stay there until you
delete them") is now true, and the privacy policy can link to a help
page
## Linked Issues or Issue Description
**Subsystem affected**
Connections (tool gateway audit tables, run logs, run events) and the
Connectors UI.
**Problem or motivation**
Paperclip stores content from a connection's tool calls in several
places and gives users no way to delete it:
- `tool_invocations.arguments_summary/result_summary` and
`tool_call_events.arguments_summary/request_summary/result_summary` keep
the first ~4,000 characters of each redacted value
(`summarizeToolValue`, `tool-access-policy.ts` `redactArguments`). No
retention limit.
- `tool_action_requests` keeps `canonical_arguments_summary`,
`preview_markdown` and `signed_arguments`. The linked approval card
(`issue_thread_interactions`) keeps copies in `payload.detailsMarkdown`,
`payload.toolAction.previewMarkdown`,
`payload.toolAction.argumentsSummaryJson` and
`result.toolAction.resultSummary`.
- `tool_mcp_app_views` keeps `tool_input` and `tool_result` for MCP app
views.
- Run transcripts hold full tool output: the run log (`local_file` with
an optional S3 mirror, or `s3_segments`/`s3_archive` object journals),
`heartbeat_run_events.message/payload`, and
`heartbeat_runs.stdout_excerpt/stderr_excerpt`. Neither run-log store
had a delete path.
- Connection removal (`removal.ts`) archives the connection and revokes
credentials. It deletes no content.
**Proposed solution**
A durable, resumable, company-scoped deletion request per connection.
See "Design" below.
**Alternatives considered**
- Delete synchronously in the request. Rejected: an interrupted request
leaves partial state that needs a person to retry it.
- Delete only the run events that mention the connection. Rejected: tool
output also reaches the model's later messages, so the safe unit is the
whole run transcript.
- Delete the audit rows. Rejected: counts, outcomes and hashes feed the
activity history and usage counts.
## Design
**Rule.** The job never drops a reference to content it has not
confirmed deleted. It is marked complete, and writes the
`tool_connection.content_deleted` activity entry, only after every step
succeeded. A step that cannot confirm its work throws. The request then
stays pending, the UI says "Paperclip keeps trying until it finishes",
and the worker retries it.
**Store inventory.** Every place that stores tool arguments, results or
previews. I found them from `summarizeToolValue`, `redactArguments`,
`writeAudit`, and every column or JSON field named like summary,
preview, arguments, result, toolInput or toolResult.
| Store | Content | Cleared by this PR | Proof |
| --- | --- | --- | --- |
| `tool_invocations.arguments_summary`, `result_summary`, `progress` |
Redacted request/result preview, provider progress | Yes: `summary`
becomes `""` with `contentDeletedAt`; `progress` null; size and sha256
kept | content-deletion test 1 |
| `tool_call_events.arguments_summary`, `request_summary`,
`result_summary` | Same previews | Yes, same | test 1 |
| `tool_access_audit_events.details.argumentsSummary` (`writeAudit`) |
Argument preview | Yes, same; decision, ids, size, hash kept | test 1 |
| `tool_action_requests.canonical_arguments_summary`,
`preview_markdown`, `signed_arguments` | Approval preview and arguments
| Yes, for settled requests | test 1, card retry test |
| Approval card in `issue_thread_interactions`:
`payload.detailsMarkdown`, `payload.toolAction.previewMarkdown`,
`payload.toolAction.argumentsSummaryJson`,
`result.toolAction.resultSummary` | Approval preview, arguments,
executed result | Yes. Cards are found from every settled request of a
batch, in the same transaction | test 1, card retry test |
| `heartbeat_runs.context_snapshot` and `agent_wakeup_requests.payload`
(`toolAction(s)[].resultSummary`,
`paperclipAgentMessage.untrustedToolResults[].resultSummary`) | Approved
action result delivered to the agent | Yes, for settled runs and settled
wakes | test 1 |
| `tool_mcp_app_views.tool_input`, `tool_result`, `model_context` | App
view input, result, shared context | Yes: rows deleted | test 1 |
| Run log: object journal, archive journal, original `local_file` log
and its legacy S3 mirror | Full transcript | Yes, every copy, for each
settled run that called the connection or received an approved result |
run-log-store tests, test 1 |
| `heartbeat_run_events.message`, `payload` | Transcript events | Yes,
except structural events (lifecycle, terminal, presentation, usage,
compaction, session/turn start, `native.*`) | test 1 |
| `heartbeat_runs.stdout_excerpt`, `stderr_excerpt`, and every string
`summary`, `result`, `message`, `text`, `output`, `content` or `body` at
any depth of `result_json` (including `nativeResult`, `acceptedResult`,
`semanticResult` and their `continuation`) | Output excerpts and final
answer | Yes; schema and control fields kept | test 1, canary |
| `approvals.payload` (formal board approval) | Tool name, risk and
hashes only | Not needed: no content | — |
| `tool_mcp_tasks` | Request hash and sealed provider task id; no result
| Not needed: no content | — |
| `tool_mcp_elicitations.questions` and their question cards | Questions
the provider asked a person | No. These are provider prompts, not tool
results, and the answered question stays in the task. Follow-up if
wanted | — |
| `tool_mcp_skills`, `tool_mcp_skill_files`, `tool_catalog_entries` |
Skills and tool descriptions the provider publishes | No. These are
provider definitions, not data read through tool calls | — |
| Transcripts of later runs that read an app view's `model_context` |
Shared app context | No. Nothing records which runs read it. The help
page says so. Follow-up | — |
| `issue_comments.body`, issue documents (including continuation
summaries), files, chat messages | Task content written by agents or
Paperclip | No, by design. The user removes it by editing or deleting it
or the task. The confirmation says in bold that anything agents wrote
into tasks is not removed, and so does the help page. The canary
excludes `issue_comments.body` by name for this reason | canary asserts
it is kept |
| Raw provider traces | Full provider traffic | No. Admin opt-in,
encrypted, expire within 24 hours | — |
| Feedback exports already sent | Run logs the user chose to share | No.
Already outside Paperclip | — |
| Content of runs of a deleted task or company | Same stores | No.
Follow-up | — |
**Durable request (schema change).** I added one table,
`tool_connection_content_deletions`, and migration `0490`. I followed
`doc/DATABASE.md`: edited the schema, exported it from
`packages/db/src/schema/index.ts`, ran `pnpm db:generate`, and ran the
migration drift and compatibility tests. The classifier marks it
`online_compatible` (a new table and its indexes only). No existing
table can hold a durable per-connection request with a cutoff time,
cursor, counts and retry state. Without it, an interrupted deletion
could resume only when a person pressed the button again.
**Resume after restart.** The worker uses the shared
`registerDeadlineWork` lifecycle on a new
`tool-connection-content-deletion` queue. The request insert signals the
queue in the same transaction. At server start the worker runs once,
`next()` finds any pending row, and processing continues from the saved
phase and cursor. Nobody presses anything. `next()` returns a deadline
only while a pending row exists, so an idle instance sends no recurring
query. A failure records `attempts` and `last_error` and retries after
30 seconds.
**One run selection.** `runsWithStoredContentSql` selects the runs whose
transcripts are deleted: runs that made a tool call or tool-call event
through the connection, and runs whose context lists one of the
connection's approved action requests, limited to runs that still hold a
log, an excerpt, result text or a content event. The dialog count, its
in-progress warning and the job's `runs` phase all use this one
function.
**Phases.** `invocations`, `events`, `auditEvents`, `appViews`,
`deliveredResults` (50 action requests per page; matching runs and wakes
are locked with `FOR UPDATE` before their documents are rewritten),
`runs` (the shared selection, 50 run ids per page, then one run at a
time), `done`.
**Canary test.** `tool-connection-content-deletion-canary.test.ts` puts
a unique marker in a tool call's arguments and result, an approved
action with a preview, a result delivered to a later run and its wake, a
native run that quotes the marker in its answer and events, a run log
migrated from a local file with an S3 mirror to an object journal, an
MCP App view and the audit tables. It checks that the marker is first
present in the database, the object store, the legacy mirror and the log
files. After the deletion it scans every `text`, `varchar`, `json` and
`jsonb` column of every table found through `information_schema` (no
hand-written table list), every object in both stores and every file
under the log directory. Any hit fails with the table and column or the
key. It runs three times: a complete deletion, a deletion interrupted
twice by storage failures and resumed, and two deletions for different
connections on one task at the same time. The only exclusion is
`issue_comments.body`, named with its reason.
**Batching and time budget.** Tool rows are read in pages of 200 with
keyset pagination on the existing `(company_id, created_at)` indexes,
bounded by the request time. Each page's writes and the saved progress
commit in one transaction. Counts are therefore exact, and a crash
repeats at most one page, idempotently. Runs advance one at a time from
the shared selection. Each 1,000-event page is its own step. Log removal
is its own step and runs before its progress transaction, so object
storage calls never run inside a database transaction. The 10-second
budget is checked after every step. Nothing changes on the agent boot or
tool-call path.
**In-flight work.** A run counts as settled only when its status is
terminal, `finished_at` is set, cost accounting is done, and no native
finalization is open. Calls, approvals, events, app views, delivered
results and transcripts tied to other runs or wakes are skipped and
counted. The confirmation says "N runs are still working and will be
skipped. Delete again after they finish."
**Permissions.** Board users with configure access (connection manager
or creator) can delete, like `DELETE /tool-connections/:id`. Agents,
viewers and other companies are refused. The activity entries
`tool_connection.content_deletion_requested` and
`tool_connection.content_deleted` hold counts only.
**UI reads.** The dialog loads counts once when it opens. While a
deletion runs, it polls `GET
/tool-connections/:id/stored-content/deletion`, a single indexed row
read, every 2 seconds. Polling stops when the job completes or the
dialog closes. The Delete button is not gated on counts.
## What Changed
- `packages/db`: new `tool_connection_content_deletions` table and
migration `0490` (online compatible: new table and its indexes only).
- `packages/shared`: `ToolConnectionStoredContent`,
`ToolConnectionContentDeletion` and counts types; optional
`contentDeletedAt` on `ToolRedactedValueSummary`.
- `server`: `RunLogStore.remove`, which deletes every copy of a run's
log (`composeRunLogStore`); `content-deletion.ts` service; deadline
worker wired into startup and pre-drain; `GET
/tool-connections/:id/stored-content`, `POST` and `GET
/tool-connections/:id/stored-content/deletion`, and `DELETE
/tool-connections/:id?deleteStoredContent=true`; OpenAPI entries.
- `ui`: "Delete data" row on the connection's Permissions tab with a
confirmation that shows counts, skipped runs, and "This cannot be
undone"; "Also delete data read through this connection" checkbox in the
Connectors removal dialog.
- Docs: user help
`docs/guides/board-operator/deleting-connection-data.md` (for the
privacy policy link) and an API note in
`doc/connections/CONNECTOR-PLAYBOOK.md`.
- `tests/runner-e2e/source-sdk-public-inputs.json` refreshed for the
shared type change.
`server/src/routes/tool-access.ts` (3,010 lines) and
`ui/src/pages/apps/Browse.tsx` (1,722 lines) grow by about 40 and 20
lines. The routes reuse that file's connection access checks; the dialog
reuses the existing removal confirmation. The new logic lives in its own
files.
## Verification
- `server/src/__tests__/tool-connection-content-deletion.test.ts`
(embedded PostgreSQL): content is gone from every "Yes" store in the
inventory; rows, hashes and sizes stay; another connection and another
company are untouched; a running run is skipped and is deleted after it
settles; a storage failure leaves the request pending with `retrying`,
and the next pass completes it; an approval-card update that fails once
is cleared on the retry; an unrecognised log backend keeps the pointer,
keeps the job pending and writes no completion entry; a zero time budget
yields after every step and still finishes; a repeat request returns the
pending one; agents and outsiders are refused; removal with
`deleteStoredContent=true` queues the deletion.
- `run-log-store.test.ts` and `object-run-log-store.test.ts`: `remove`
deletes local and mirrored logs and object journals, removes a migrated
run's original local file and mirror with its journal, refuses an object
log when object storage is not configured, and is idempotent.
- `ui/.../StoredContentDeletion.test.tsx`: confirmation text, counts,
skipped-run warning, and the request.
- Red: these tests fail on `master` because `remove`, the service and
the routes do not exist. Green on this branch.
- Ran: the tests above, `openapi-routes`,
`server-startup-feedback-export`, `tool-connection-removal`,
`packages/db` migration drift and compatibility tests, all
`ui/src/pages/apps` tests (950), `pnpm check:token-gates`, server and UI
`tsc --noEmit`. `pnpm verify` exits 2 (global CI input changed), so
required CI supplies broad coverage.
## Risks
- Deleting is permanent by design. The confirmation says so.
- A `local_file` log written on another replica's local disk stays on
that pod's disk until the pod is replaced; its S3 mirror is deleted. New
logs on S3-configured instances use object journals, which are deleted.
- On versioned buckets, earlier object versions remain until the
bucket's lifecycle policy removes them (the existing journal delete path
does not remove versions).
- Not in this PR (follow-ups): the "No" rows of the inventory (except
agent-written task content, which is out of scope by design), mainly the
same deletion when a task or a company is deleted, and later runs that
read an app view's shared context.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use and code execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
55b692d532 |
feat(telemetry): send Cloud server telemetry directly to PostHog (#22006)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server sends first-party telemetry events through a collector. The collector anonymizes the events and copies them into PostHog. > - Paperclip Cloud is our own service. It needs real IDs (workspace, user, agent, task) so that a person can go from a PostHog event to the exact workspace and task. > - The collector path hashes these IDs. It also cannot link a server event to the browser session that caused it. > - This pull request adds a server-owned direct sender for Cloud only. Self-hosted installs keep the collector path with no change. > - The benefit is real IDs and session links on Cloud, and no change to self-hosted privacy. ## Linked Issues or Issue Description **Subsystem affected** Server telemetry (`packages/shared/src/telemetry`, `server/src/telemetry.ts`). **Problem or motivation** All Cloud telemetry goes through the collector. The collector hashes the install ID, agent IDs and task IDs. So, on Cloud, you cannot find the workspace, user or task of an event. Server events also cannot show in a browser session replay. **Proposed solution** On a Cloud-managed server that has `PAPERCLIP_POSTHOG_PROJECT_TOKEN`, `PAPERCLIP_CLOUD_STACK_ID` and `PAPERCLIP_POSTHOG_ENVIRONMENT`, send each telemetry batch straight to PostHog with `posthog-node`. Use the same event shape that the telemetry backend loader makes today, and add the real IDs. Keep every other install on the collector. **Alternatives considered** Send real IDs through the collector. We did not do this, because the collector exists to anonymize self-hosted data. **Roadmap alignment** This is Cloud analytics infrastructure. It does not change a core product feature. ## What Changed - Add `server/src/services/cloud-posthog-config.ts`. `resolveCloudPostHogConfig()` returns `null` unless the instance is Cloud-managed and has a token and a stack ID. `isPostHogDirectDeliveryEnabled()` also requires an environment and the kill switch on (`PAPERCLIP_POSTHOG_DIRECT` is not `0`). - Add a narrow `TelemetryTransport` seam to `TelemetryClient` in `packages/shared`. With a transport, each flushed batch goes to the transport and never to the collector. Without one, the client does what it did before. A transport can capture per-event context at track time. The context does not go into the collector envelope. - Typed helpers hash `task_id` unless the transport sets `hashesPrivateRefs: false`. Private `source_ref` values (for example a private repository name) are customer content, so they stay hashed on every path, Cloud included, with `source_ref_hashed: true`. - Add `server/src/services/posthog-direct/`: - `event-shape.ts` builds PostHog messages with the loader's event names, metadata keys and key order, the deterministic event uuid formula, and the unknown-enum fallback. It drops events that the backend quarantines (unregistered names). - `enum-fallbacks.generated.ts` holds the backend fallback rules. Enum values come from the shared generated contract. - `transport.ts` loads `posthog-node` with dynamic `import()` only on the first Cloud send. - `request-context.ts` is Cloud-only Express middleware. It reads only `X-POSTHOG-SESSION-ID` and `X-POSTHOG-WINDOW-ID`. It ignores `X-POSTHOG-DISTINCT-ID`. The acting user comes only from the authenticated request actor. Context stops when the response finishes, so later background work does not use that user. - `server/src/telemetry.ts` selects exactly one sender per process. On start, the Cloud sender sends `$groupidentify` (group `workspace`, key = stack ID) and one `cloud.workspace_started` event with `install_hash` (sha256 of the raw install ID). Both go straight to `posthog-node` and are not in `PAPERCLIP_EVENTS`. - `distinct_id` is the acting user ID, else `workspace:<stackId>`. Events without an acting user keep `$process_person_profile: false`. GeoIP stays off on all server events. - Shutdown flushes the client and then shuts down `posthog-node`. - Add `scripts/generate-posthog-direct-fixtures.mjs`. It runs the telemetry backend's own ingest and loader on one envelope and writes the golden fixture and the fallback table. The script header tells how to regenerate them. - Update `packages/shared/src/telemetry/README.md` with a "Cloud Direct Delivery" section about the privacy boundary. - Add `posthog-node@5.38.2` to `server/package.json`. This PR does not change `pnpm-lock.yaml`, because the lockfile refresh workflow owns it. Intended differences from the loader output on Cloud: `distinct_id` (user or workspace, not the HMAC install ID), raw `agent_id`, raw `task_id` (private `source_ref` stays hashed), and these added properties: `stack_id`, `company_id` and `user_id` when known, `environment`, `$groups`, `$session_id` and `$window_id`. The uuid formula is the same. The uuid is different only when the dimensions are different (raw `task_id`). ## Verification - `pnpm verify` returned exit 2 (a global CI input changed: the new script). I ran focused checks. Required CI supplies the broad coverage. - Focused tests pass: `packages/shared` `src/telemetry` (6 files, 42 tests). Server: `posthog-direct-telemetry`, `posthog-direct-event-shape`, `cloud-posthog-config`, `shared-telemetry-events`, `heartbeat-status-preserving-telemetry`, `issue-thread-interactions-telemetry`, `native-run-finalizer-telemetry`, `agent-task-run-telemetry`, `error-handler` (9 files, 115 tests), and `server-startup-feedback-export` (133 tests). - `tsc --noEmit` passes for `packages/shared`, `server` and `cli`. - New tests: - Config matrix: not Cloud, no token, no stack ID, no environment, kill switch, browser and replay flags. - Self-hosted: `posthog-node` is never loaded, the collector gets the batch, and the envelope bytes are the same as before (with `task_id` hashed). Tracing headers are ignored. - Cloud: a private `source_ref` stays hashed and the raw repository name never reaches the sender. - Cloud: one batch goes only to PostHog. Real IDs, `$groups`, `environment` on every event. `cloud.workspace_started` reaches the PostHog sender with `install_hash`. No other event has `install_hash`. - Golden: our builder output equals the backend loader output, except the intended differences above. Key order is the same. - `distinct_id`: user, agent and anonymous requests. A spoofed `X-POSTHOG-DISTINCT-ID` never changes `distinct_id` or `user_id`. A bad session header is dropped. Work after the response ends does not keep the user (red without the response-end guard, green with it). - Kill switch and a missing environment both fall back to the collector. - Local run with a real server and a local capture server (no real PostHog project): - Cloud mode (authenticated, local Postgres, `PAPERCLIP_POSTHOG_HOST` set to the capture server): the capture server got `$groupidentify`, `cloud.workspace_started`, `server.started` and `goal.created` on `/batch/`. `goal.created` from a Cloud tenant user had `distinct_id` = that user, `user_id`, `company_id`, `$session_id`, `$window_id`, `$groups` and no `$process_person_profile`. The spoofed distinct ID header was not used. Nothing went to `/ingest`. `SIGTERM` ran the `telemetry_flush` step and sent the queued events. - Self-hosted mode (same token variables but no Cloud signal): the capture server got only `/ingest` with the normal envelope. Nothing went to `/batch/`. - Runtime dependency note: `posthog-node` loads with dynamic `import()` only on Cloud with direct delivery on. The load happens after startup, outside the boot path. The unit test mocks the package to count loads. The local run used the real package. ## Risks - Medium. This is a telemetry change, so a privacy review is requested. Self-hosted output does not change. The direct path is off unless all three Cloud variables are set, so a merge does nothing until fleet config sets them. - Cloud events with an acting user create PostHog person profiles keyed by the Paperclip user ID. This is intended under the Cloud terms. - The fallback table can drift if the backend adds a new normalization rule. A test fails when a rule does not match the shared enum contract. Enum values come from the shared contract, so new enum values do not drift. - `posthog-node` keeps network failures in its own queue and drops events that PostHog rejects. The client does not resend these, so events are not sent twice. - `pnpm-lock.yaml` is not in this PR. The lockfile refresh workflow adds `posthog-node` after merge. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, local server runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
3afb2c215b |
fix(server): name the blocking database sessions whenever a run's event writes stall (#21956)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server stores the events that a runner sends for each run, and it records the identity of the runner process > - Sometimes the server stops storing a run's events for minutes, and the server log stays quiet > - In two restart-chaos cases, the identity write of the run started and then did not commit for about 197 s and 480 s > - The existing database report only fires when the reattach phase ends at its cap, so it did not fire for the cold-start case > - This pull request logs the pending writes and the blocking database sessions whenever any run write stays pending for 30 s > - The benefit is that the next stall names the session and the query that hold the run, so the fix can target one proven holder ## Linked Issues or Issue Description Refs #21946 **What happened?** Restart-chaos waves on master showed two stalls in event storage. 1. Cold start of an OpenCode run, with no restart. The runner emitted `session.started` at 22:13:23.7Z. The server stored it at 22:16:40.97Z, about 197 s later. The second `native.process_identity_recorded` write of the run started at 22:13:23.706Z, and `runner.turn.submit` took 197270 ms. A second wave showed the same shape (about 191 s). The server log had nothing in that window. 2. After a hard kill, every API GET timed out from 22:35:24Z to 22:43:23Z, server CPU was 0%, and no run events were stored. The process identity transaction of the run began at 22:35:22.6Z and committed at about 22:43:23Z. **Expected behavior** When the server cannot store a run's events, the server log names the cause: which run writes are pending, and which database sessions hold or wait for locks. **Steps to reproduce** 1. Start a transaction that locks a `heartbeat_runs` row (`for update`) and stays idle. 2. Record a process identity for that run (`persistHeartbeatRunProcessMetadata`). 3. Before this change, nothing is logged while the write waits. After this change, a warning names the idle holder and the waiting statement after 30 s. The new database test does this with a 300 ms threshold. **Relevant logs or output** Example warning from the new test: `runner event intake stalled; pending run writes, database sessions and lock waits` with `operations: [{stage: "process_identity", pendingMs: 301}]` and `waits: [{state: "idle in transaction", query: "select id from heartbeat_runs where id = ? ... for update"}, {waitEventType: "Lock", blockedBy: [<holder pid>]}]`. ## What Changed - New `server/src/services/run-intake-stall-watch.ts`. It keeps an in-memory list of pending run operations: event receipts (`PaperclipControlPlanePort.appendEvent`), event writes (`appendHeartbeatRunEvent`, `appendHeartbeatRunEventBatch`), and process identity writes (`persistHeartbeatRunProcessMetadata`). - When an operation stays pending for 30 s, the watch logs one warning per stalled run. The warning has: the pending operations and their age, the pending count and oldest age per run, the pool size, the database sessions per application name and state, the lock waits and holders (`pg_blocking_pids`, query without literals, cut to 160 characters), and the event-loop lag. Reports are at most one per 30 s. A second line logs when a reported operation finishes. - The watch arms a timer only while an operation is pending. It reads the database only when it reports, through the private liveness pool with a 5 s statement timeout. - The server names its main pool `paperclip:<boot id>` unless `DATABASE_APPLICATION_NAME` is set. This shows which server process owns each session after a restart. - `readDatabaseSessionWaits` moves from the reattach phase to `server/src/services/database-session-waits.ts`. The reattach phase re-exports it. - `@paperclipai/db` exports `databaseClientOptionsFromEnv` and `DatabaseClientOptions`. - `doc/DATABASE.md` describes the new log lines and the pool name. - I did not add `idle_in_transaction_session_timeout` to the main pool. Some code holds a transaction open on purpose, and a blind timeout could abort valid work. The new report names idle-in-transaction holders instead. - These are server log lines only. This change does not touch Telemetry or OpenTelemetry. ## Verification - `cd server && npx vitest run src/services/run-intake-stall-watch.test.ts` (5 tests). The database test holds a run row in an idle transaction, starts a process identity write, and checks that the warning names the holder pid, its query, the waiting statement, and the pool name. It is red without the tracking in `run-lifecycle.ts` and `heartbeat-run-events.ts` (`expected [] to have a length of 1`), and green with it. - Unit tests with fake timers check one report per stalled run, the 30 s throttle, the clear line, and that no timer stays armed when nothing is pending. - `cd server && npx vitest run src/services/controller-process-identity.test.ts` checks the pool name and the `DATABASE_APPLICATION_NAME` override. - `npx tsc --noEmit` passes for `server` and `packages/db`. - `pnpm verify` ran locally on the affected tests. ## Risks - Low risk. Each run event adds one map insert and one map delete. No database work happens while writes finish within 30 s. - The main pool `application_name` changes from `paperclip` to `paperclip:<boot id>`. A dashboard that filters on exactly `paperclip` must use a prefix match, or the operator can set `DATABASE_APPLICATION_NAME`. - A report reads `pg_stat_activity` on the liveness pool. A report during a full database stall can fail; the warning then has `databaseReadError` and still lists the pending operations. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used as a coding agent with tool use (shell, file edits, local test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
c66530da63 |
fix(server): release settled native run process identity at graceful shutdown (#21689)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A server process records the local process of each native run in
`heartbeat_runs` (`process_pid`, `process_group_id`,
`process_started_at`, `process_host_hash`)
> - Only a server in the same PID space can prove that such a process
stopped. `proveLocalOwnerGone` does not accept a recorded process group
from another container
> - Each deploy replaces the server process. Settled native runs that
the old server ran keep their process fields, and no later server can
release them. Recovery sweeps had to learn to skip these rows one by one
> - The stopping server is the one process that can check its own PID
space. After its graceful drain, it can see that a settled run's process
exited
> - This pull request adds a graceful-shutdown step that releases the
process identity of those runs, with the same rules that the running
server already uses, in one candidate read and one update
> - The benefit is that a deploy no longer adds stranded process records
for the runs that the stopping server could prove stopped, and a
successor on the same host has nothing to release at startup
## Linked Issues or Issue Description
Refs #21590
**What happened?**
After a graceful shutdown, settled native runs (`failed`, `cancelled`,
`timed_out`, `interrupted`) that the stopping server ran keep
`process_pid` and `process_group_id`. The successor runs in a new
container with a different `process_host_hash`. It cannot prove that the
recorded process stopped, so the rows stay forever. A successor on the
same host releases them at startup with about 10 statements for each
run.
**Expected behavior**
The server that ran the process knows that the process exited. A
graceful shutdown should release these records before the server stops.
An abrupt stop should keep today's behavior.
**Steps to reproduce**
1. Run native work so that some runs settle while the server runs, or
settle during the shutdown drain.
2. Send SIGTERM to the server.
3. Count settled native runs with `process_pid` or `process_group_id`
set: the count does not go down after the deploy.
**Paperclip version or commit**
master at
|
||
|
|
cf09024292 |
fix(heartbeat): count an early native hand-off only after its suspend commits (#21572)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A server restart hands each live Runner run to the next server, so
the agent keeps working without a retry
> - At shutdown start, the outgoing server hands off native runs early,
in parallel with the rest of shutdown
> - The early hand-off put a run in the "handed off" set before its
suspend finished, and the drain counted every run in that set as
suspended without waiting
> - The early hand-off shares the 20 s pre-drain budget, and one Runner
detach can take 20-30 s, so the process could exit with the suspend
unfinished
> - Then the finalization lease stays live, the next server gets no
hand-over signal, and the run waits for lease expiry or the reaper
> - This pull request makes the drain wait for each in-flight early
hand-off, and fall back to its own hand-off when the early one failed or
did not finish
> - The benefit is that a run counts as handed over only after its
hand-over is committed, so the next server can always reach it at once
## Linked Issues or Issue Description
**What happened?**
A passing zero-downtime swap test showed a latent race in the shutdown
hand-off. `handOffNativeRunsForShutdown` added a run to
`nativeRunsHandedOffEarly` before `suspendNativeRunForShutdown`
finished. `drainRunningRunsForShutdown` then reported every run in that
set as `restartSuspended` without awaiting it. Because the early
hand-off shares the 20 s pre-drain budget and a Runner detach can wait
20-30 s, the process can exit before the lease release commits. Also, a
timed-out `prepareHotRestartShutdown` could still write the hot-restart
snapshot and detach sessions after the fallback drain started. In the
same test, swap 1's early hand-off took 2.08 s, and no log showed which
phase was slow.
**Expected behavior**
A run counts as handed over only after its suspend commits. If the early
suspend fails or does not finish, the drain hands the run off itself. A
timed-out hot-restart preparation does no further work after the drain
starts. The hand-off logs its phase timings.
**Steps to reproduce**
1. Start a native Runner run on a remote sandbox.
2. Send SIGTERM while the Runner detach is slow (or fails).
3. The drain reports the run as restart-suspended while its suspend is
still pending, and the process can exit before the lease release.
**Paperclip version or commit**
`master` at
|
||
|
|
e977c3b922 |
feat(native-runtime): end the wait for the provider's own turn when a person comments (#21455)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Core runs Claude Code through Paperclip Runner. Claude can start a shell command in the background and end its turn. When the command finishes, Claude Code starts a cycle of its own to continue. > - Today the run ends when the turn ends. The command stops, and the run fails with `native_finalization_missing`. > - The Runner part of this work is complete in #21316 and the pull requests below it: when a controller sends `providerStartedTurns` on `turn.start`, the Runner keeps Claude while a background shell is running, and gives the cycle of Claude to the controller as a turn that the provider started. > - Part 2 gave the consumer a wait with a limit and two inputs: a hook that ends the wait early, and a start time. > - This pull request gives both from the server for a native ACPX Claude run. A new comment from a person ends the wait. The start time comes from an event that is already saved, so a restart of Core does not make the wait longer. > - The benefit is that a person who comments does not wait up to 10 minutes for the agent to see it. ## Linked Issues or Issue Description No public issue exists. The issue is described here. This pull request is one of four parts on the Core side. They are stacked on #21316: 1. The driver admits a turn that the provider started. 2. The consumer waits for that turn, for a limited time. 3. The server ends the wait when a person comments, and keeps the clock across a restart. 4. The transport sends the opt-in. Only this part changes what a production run does. This is part 3. Its base branch is `feat/core-provider-turn-consumer`. Refs #19912 (design). **What existing behavior does this improve?** Nothing visible yet. What Core gives to the consumer of a native ACPX Claude run. **Subsystem affected** Server: `server/src/services/native-runtime`. **Current behavior** - A comment from a person during a run is queued as a deferred wake. The next run delivers it. This does not change. - Core has no wait for a provider-started turn. **Proposed behavior** - For a native ACPX Claude run, Core gives `executeNativeSession` the option `providerTurnWait`. - One delivery worker does the person-comment check for all runs that wait. The server registers it at startup with the shared delivery work coordinator, as it registers the cost accounting worker. The worker listens to the existing notification for deferred wakes. On each notice it runs one query for each run that waits: a comment from a person on this issue, made after the start of the run, that a deferred wake of this agent names. A comment that the query finds ends the wait of that run. The run then takes the path of today, and the next wake delivers the comment as today. - If a database read fails, that run stays owed a check, and the coordinator runs the worker again after 5 seconds, as for each other delivery worker. A read that fails for one run does not stop the check of the other runs. When no run is owed a check, the worker has no deadline and does no query. - The start time of the wait is the time of the first saved `turn.completed` event of the run that has `awaitingProviderTurn`. With no such event it is empty. - Each query has the company id in its condition. **Reason and benefit** - The comment window starts at the start of the run, not at the start of the wait. A comment made during the prompt turn must not wait for the limit. ## What Changed - `server/src/services/native-runtime/native-provider-turn-wait.ts` (new, 109 lines) and 4 lines in `native-session-executor.ts`. - `server/src/index.ts` (4 lines): the registration of the worker, its stop at server close, and its place in the list of delivery workers that must be ready. `delivery-work-notifications.ts` (1 line): the queue key of the worker. The key needs no database change, because each topic uses the one shared channel. - `server/src/vendor/paperclip-runner/index.ts`: one type export. - Tests: `native-provider-turn-wait.test.ts` (new, 252 lines) and one case in `native-usage-accounting.test.ts` (25 lines). ## State saved by the previous release - No new persisted state and no schema change. The start time is read from `heartbeat_run_events`. - **A run that the previous release started:** it has no waiting terminal, so the start time is empty. - **Core restarts during a wait:** the new process reads the start time, so the limit is the original one. The consumer then continues the wait (#21454), and the worker checks the comments again from the start of the run, so a comment that was made before the restart still ends the wait. - **Rollback of Core during a wait:** the previous Core stops at the waiting terminal and finalizes the run as `native_finalization_missing`, as today. ## Verification At the head of this branch: - `npx tsc --noEmit -p tsconfig.json`, `npx tsc --noEmit -p tsconfig.test.json` and `pnpm run build:typescript` in `packages/paperclip-runner`; `node scripts/verify-runner-vendor-dependencies.mjs` and `pnpm typecheck` in `server`; `node --test scripts/__tests__/runner-release-crate-cache.test.mjs`: all passed. - `native-provider-turn-wait.test.ts`, 10 tests on embedded Postgres with the real coordinator: a comment from a person in a deferred wake ends the wait; a comment from an agent, a comment before the run, and a comment of a different company do not; the start time comes from the first waiting terminal, and a run of the previous release has none. - The 6 new tests: a read that fails one time is tried again after 5 seconds and finds a comment that is already saved, with no new notice; a notice during a read that fails gives a new check; a read that fails for one run does not hold back a second run, and each run ends only for its own comment; a run that starts to wait during a check is checked; a wait that is aborted is removed, nothing checks it again and no deadline stays; with no run owed a check there is no deadline and no query in one hour of test time. - The test helper now registers the wake intent before it writes the deferred wake, in the same transaction. This is the order of each production writer (`modules/wake-queue/adapters/postgres.ts` and `services/heartbeat/queue.ts`). Production code did not change for this. - Mutation checks: a read that fails clears the owed check (3 tests fail); an aborted wait is not removed (the abort test fails). - Also run, unchanged and green: `delivery-work-coordinator.test.ts`, `budget-enforcement-worker.test.ts`, `server-startup-feedback-export.test.ts`, `startup-timeline.test.ts`. The 7 files together: 1,699 passed, 7 skipped. - `native-usage-accounting.test.ts`: a prompt turn and the provider-started turn after it are each counted one time, with the events as the executor receives them. The cost of the provider-started turn is the session total in the cumulative channel, and it is not added two times. The accounting code is not changed. Not run: a run through the executor with a database; the full suite. CI supplies the broad coverage. ## Risks - **One more query at each start of a native ACPX Claude run,** on `heartbeat_run_events` by company and run. - **The worker is registered in each server process, and it is idle with no wait.** A notice with no run that waits does no query. No wait starts before part 4. - **After an abort during a retry, the 5 second retry of the coordinator runs one more time.** It finds nothing owed and does no query. The coordinator has no call to cancel the retry of one worker. - **A process that does not register the worker does no comment check.** The server registers it with no condition at startup, and native runs execute only in that process. Only a unit test can be in this state. - **A comment ends the wait but does not interrupt a turn that is running.** That is the rule of part 2. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`) in Claude Code planned the change and the split, reviewed the diff, and ran a part of the verification again. - Claude Opus 5.5 (`claude-opus-5-5`), as a Claude Code subagent with tool use (shell, file edits, local test runs), wrote the change and the tests and cut this slice from the full change. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
3ae7aba643 |
perf(db): reuse dedicated connections instead of opening one per call (#21567)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server uses `withDedicatedDbConnection` (packages/db) when an action must hold one database session outside the main pool: advisory transaction locks, lock keepalive, and `pg_settings` reads for workspace lifecycle, finalization, source locks, skill caches and status cards > - Each call built a new postgres.js client, opened a new physical connection (TCP, TLS and startup through the pooler), built a new drizzle client with the full schema, and then ended all of it > - On staging the dedicated application name shows about 21 connection setups per second, and the main thread is pinned (event-loop utilization 0.965) > - This pull request keeps a small set of idle single-connection clients per owner and reuses them > - The benefit is that a dedicated action costs about 0.2 ms of main-thread time instead of 4 to 8 ms, with the same exclusive-session guarantee ## Linked Issues or Issue Description **What existing behavior does this improve?** `withDedicatedDbConnection` opens and closes one new database client and one new physical connection for every call. **Subsystem affected** Database client (`packages/db`) and the server services that take dedicated advisory-lock sessions. **Current behavior** Each call builds a new postgres.js client, opens a new connection (TCP, TLS and startup through the pooler), and builds a new drizzle client with the full schema. On staging (Neon statement statistics, 326 s window) `SET application_name='paperclip-workspace-finalization-lock'` ran 21.4 times per second. Only this factory uses that application name. In a 30 s staging CPU profile, drizzle `relations.ts` was 2.2% inclusive, `n:net` 5.8% and postgres.js construction 0.3%. Part of that is this per-call setup. **Proposed behavior** A caller still gets exclusive use of one session for the whole action. The session and its drizzle client are reused by later actions, so the per-call cost is the action's own queries, not a full connection setup. **Reason and benefit** The server main thread is saturated on staging (event-loop utilization 0.965). Reuse removes about 5 ms of main-thread work per call. **Breaking changes** None. The exported function keeps its signature. `closeDedicatedDbConnections` and `DEDICATED_CONNECTION_IDLE_MAX` are new exports. ## What Changed - `packages/db/src/client.ts`: each `createDb` owner now has one set of dedicated single-connection clients, shared with every private client derived from that owner (so nested dedicated actions use the same set). - An action takes an idle client (last in, first out) or opens a new one. It never waits. Callers hold advisory locks and nest dedicated actions, so a waiting queue could deadlock. - On success the client returns to the idle set if fewer than `DEDICATED_CONNECTION_IDLE_MAX` (4) are idle. Otherwise it is closed. - When the action throws, the client is closed and never reused. This covers a failed or cancelled transaction, a broken socket, and an interrupted lock wait. - When an action ends the client itself (the source-lock cancel path calls `$client.end`), the client is never reused. - Idle clients keep the owner's idle timeout and max lifetime, so postgres.js still closes idle physical connections after `DATABASE_IDLE_TIMEOUT_SECONDS`. - New `closeDedicatedDbConnections(db, timeoutSeconds)` closes the idle set and stops retention. Ending the owner `$client` does the same. A later action still works with a connection that closes on release. - `server/src/index.ts`: shutdown closes the dedicated set next to the other private pools. - `doc/DATABASE.md`: documents the idle cap, the "not an admission limit" rule, and the transaction-scoped session state contract. - Session state audit: every current caller uses `pg_advisory_xact_lock` / `pg_try_advisory_xact_lock` and `set_config(..., true)`. No caller uses session advisory locks, `SET` without `LOCAL`, `LISTEN`, or temporary tables. A reset round trip per call is therefore not added. ## Verification - New `packages/db/src/dedicated-connection-pool.test.ts` (embedded PostgreSQL, 10 tests). - Red on `origin/master`: 200 sequential calls used 200 distinct sessions (`expected 200 to be 1`). All 10 tests failed. - Green with this change, 3 runs in a row: 200 sequential calls use 1 session; 7 concurrent callers each hold a different session for the whole transaction, and afterwards at most 4 stay open and later calls reuse them; a failed transaction's connection is closed; a terminated socket during the action is not reused; an idle connection terminated by the server reconnects; a client the action ended is not reused; transaction-local settings do not leak; shutdown and owner `$client.end` close every retained connection; nested actions do not block. - Focused suites pass: packages/db `work-signals`, `prepared-queries-postgres`, `response-capture-postgres`, `checkpoint-pool`, `client`, `client-options`, `client-teardown-registry`, `postgres-connection-recovery`, `client-disconnect-replay` (132 tests). Server suites that use dedicated connections: `skill-package-store`, `status-card-evaluation-work` (updated: it now proves advisory locks are released, since sessions are reused), `agent-instruction-working-copies`, `workspace-recovery-retry-copyback`, `native-execution-lease-batch`, `company-skills-empty-runtime`, `decisions-service`, `legacy-controller-lease`, `server-startup-feedback-export` (467 tests). - `pnpm verify --explain` selects 2779 files for a `packages/db/src/client.ts` change. Per AGENTS.md I ran the focused checks above and let CI give the broad coverage. - Benchmark: 200 calls of `withDedicatedDbConnection(db, d => d.transaction(tx => tx.execute(select pg_try_advisory_xact_lock(42))))` against local embedded PostgreSQL 18, 10 warm-up calls, 5 alternating before/after process pairs per mode. Main-thread time is `eventLoopUtilization().active`. | Mode | Variant | Wall ms/call (median, range) | Main-thread ms/call (median, range) | | --- | --- | --- | --- | | TLS | before | 35.55 (32.93–37.52) | 5.18 (4.68–7.95) | | TLS | after | 0.19 (0.17–0.21) | 0.18 (0.17–0.20) | | Plain | before | 18.39 (17.91–20.53) | 3.65 (3.58–4.30) | | Plain | after | 0.17 (0.13–0.18) | 0.13 (0.12–0.14) | At the staging rate of about 21 calls per second, this removes roughly 100 ms of main-thread work per second (TLS figures; staging CPUs may be slower). ## Risks - A retained connection that the server or network drops while idle: postgres.js reconnects on the next query when it has seen the close (tested). If the drop and the reuse race, the first query of that action fails like a main-pool query would. Before this change, a dedicated action always started on a fresh connection. - Through a transaction-mode pooler, `SET application_name=...` is a server-side parameter switch, so its count may not drop as much as the client-side connection count. Measure the deployed effect with pooler client connections and main-thread CPU, not only that statement. - Concurrency above 4 opens extra connections exactly as before; they close on release. Total retained connections per owner grow by at most 4 idle clients, which still close after the idle timeout. - An action that sets session-level state would now leak it to the next action. The contract is documented on `withDedicatedDbConnection` and in `doc/DATABASE.md`; no current caller does this. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
cefef8f11d |
perf(workspaces): inspect only hinted workspaces after a scoped commit (#21519)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The server cleans up execution workspaces whose task tree is done. A
background sweep (`terminal-workspaces`) finds these workspaces.
> - Every workspace write and every task status change sends a hint to
this sweep. Each hint starts a full rotation over every open candidate
workspace, and each candidate costs a recursive read of its task tree.
> - Runs write workspaces all the time, so with live runs the sweep
never stops, and its cost grows with the number of open workspaces.
> - On a hosted instance with 5 live runs, the task tree read ran 11.6
times per second. On a local copy with run churn it ran 81 to 101 times
per second.
> - This pull request inspects only the workspaces that a hint names or
reaches. Full rotations stay for startup, saved deadlines, failures and
hints without a scope.
> - The benefit is that a workspace change costs a few statements, not
one tree read per open workspace.
## Linked Issues or Issue Description
No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/enhancement.yml`.
**What existing behavior does this improve?**
The terminal workspace sweep (`registerTerminalWorkspaceWork` in
`server/src/services/terminal-workspace-work.ts` and
`sweepTerminalWorkspaces` in
`server/src/services/execution-workspaces.ts`).
**Current behavior**
Producers send hints such as `workspace:<id>` and `issue:<id>`. The
worker ignores the scope: each settled hint wakes the queue, and the run
starts or continues a full keyset rotation over all candidate
workspaces, 50 per page with 1 second between pages. A hint that arrives
during a rotation also requests a rescan, so a new rotation starts after
the current one. Every candidate with an open task tree costs one
recursive tree read and is skipped.
**Proposed behavior**
The worker keeps the workspace ids and task ids from scoped hints. When
the queue runs only because of such hints, it calls
`sweepTerminalWorkspaces` with `targets`. The sweep then reads only
these candidates:
- the workspaces named by `workspace:` hints;
- the workspaces whose source task is the hinted task or one of its
ancestors (the tree of a workspace is its source task and its
descendants);
- the execution workspace of each hinted task.
A targeted inspection does not move the rotation cursor or its boundary
and does not clear saved deadlines. The per-workspace checks are the
same code as in a rotation.
A full rotation still runs at startup, when a saved deadline is due,
when a hint has no scope or an unknown scope, while a rotation is in
progress, and after a failed targeted inspection.
**Reason and benefit**
Background work must not grow with the backlog of open workspaces. With
4 candidate workspaces, a hint for one task tree now reads 1 tree
instead of 4. A hint during a rotation no longer starts another full
rotation.
## What Changed
- `server/src/services/terminal-workspace-work.ts`: parse scoped hints
into targets. Run a targeted inspection when only scoped hints are
pending. Keep the full rotation for startup, due deadlines, unscoped
hints, continuation pages and failures.
- `server/src/services/terminal-workspace-work.ts` also falls back to a
full rotation when more than 50 workspaces wait for inspection. It keeps
the 1-second pause between rotation pages and runs targeted inspections
between them.
- `server/src/services/issues.ts`: moving a task also hints its former
parent. Removing a task also hints its parent and its execution
workspace. Without this, a former parent's workspace could never be
reached.
- `server/src/services/execution-workspaces.ts`:
`sweepTerminalWorkspaces` accepts `targets`. A targeted page that fills
its limit requests a full rescan. and filters the candidates to the
targeted workspaces. A targeted inspection leaves the rotation state
alone. `terminalWorkspaceWorkSettled` takes `{ targeted: true }` from
the worker, and then a scoped hint does not request a full rescan.
Direct callers keep the old rescan behavior.
- `server/src/index.ts`: pass the new option through the existing
wrapper.
- Tests in `server/src/__tests__/terminal-workspace-work.test.ts` and
`server/src/__tests__/execution-workspaces-service.test.ts`.
## Verification
- New test `inspects only the workspaces a hint reaches, however many
trees stay open`: 3 workspaces with open trees and 1 done tree whose
child task is hinted. Before the change the targeted call checks 4
workspaces. After the change it checks 1, reads 1 tree, and archives the
done workspace.
- New test `inspects only hinted workspaces after a commit, and keeps
unscoped commits and deadlines on the full rotation`: before the change
the worker never passes targets. After the change a scoped commit runs a
targeted inspection, and an unscoped commit and a due deadline still run
a full rotation.
- `npx vitest run src/__tests__/terminal-workspace-work.test.ts
src/__tests__/execution-workspaces-service.test.ts
src/__tests__/isolated-task-directory-retention.test.ts
src/__tests__/workspace-delivery-cleanup.integration.test.ts
src/__tests__/git-workspace-expiration.test.ts` (in `server/`) pass.
- New tests `falls back to the full rotation when more workspaces are
hinted than one inspection reads`, `keeps the pause between rotation
pages while hinted workspaces wait` and `hints the former parent and
workspace when a task moves or is removed`. Each fails on the first
commit and passes on the head.
- `pnpm check:module-boundaries` passes.
Local benchmark. The real server ran against PostgreSQL with
`log_statement=all`. The fixture had 1 company, 60 agents, and open
tasks scaled from 150 to 5,000. Recovery re-dispatched runs to agents
that fail at once, and every run creates or updates an execution
workspace, so the hints keep coming. Each window is 180 seconds and
starts 150 seconds after the server is ready. The machine was shared.
The table gives the rate of the task tree read (`select id, status,
completed_at, cancelled_at, updated_at from issues ... WITH RECURSIVE
issue_tree`):
| Open tasks | Before (per s) | After (per s) |
| --- | --- | --- |
| 150 | 17.56 | 0.93 |
| 1,500 | 109.83 | 5.16 |
| 5,000 | 113.82 | 4.40 |
Total statements and transactions per second depend mostly on how fast
the failing runs cycle in the fixture, so they are noisy. Before →
after: 343 → 287 at 150, 1,647 → 1,562 at 1,500, 1,530 → 1,678 at 5,000
statements/s. In that window the 5,000-task run after the change also
had 10% more run events. With no activity, the server issued 0
statements before and after the change.
## Risks
- A workspace that waits on something without a hint is no longer
inspected as a side effect of unrelated activity. The known cases keep
their path: a workspace held by an active run keeps its past-due
deadline, so the 5-minute deadline rotation inspects it again; cooldown,
compaction, Git idle and retry deadlines still start full rotations. A
delivery that changes outside Paperclip without a work product change
was already only found by a later rotation.
- A failed targeted inspection requests a full rotation, so a lost
target is recovered by the rotation.
- Low migration risk. No schema change.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
47a2c82d35 |
fix(heartbeat): reach the previous server's live runs first after a release swap (#21450)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - During a release swap, the outgoing server stops and the incoming server takes over the live runs on remote Runners > - The incoming server reached those runs late: it waited up to 60 s for the outgoing server's controller lease to lapse, re-verified each run's whole log from object storage, and deferred a reattach when the agent directory was busy. Queued starts competed with the reattaches for the same resources. > - On hosted staging, the later a live run was reached, the more likely it was to fail. Runs reached in under 150 s never failed; runs reached after 420 s almost always failed. > - This pull request hands live runs over at the start of shutdown, makes a reattach skip work that the same run already did, and holds queued starts while handed-over runs are being reached, with a 180 s cap. > - The benefit is that live runs are reached within seconds of a swap instead of minutes, so they continue instead of failing. ## Linked Issues or Issue Description **What happened?** Hosted staging, release swaps on 2026-11-13 (06:05Z to 09:45Z). For each run that crossed a swap we measured the gap from the last Runner output to the moment the new server recorded the Runner's process identity: - gap under 150 s: 0 of 9 runs failed - gap over 420 s: 14 of 15 runs failed - median gap: 180 s for runs that resumed, 470 s for runs that failed In one traced run the incoming server waited on `remote_controller_lease_live` three times (09:14:21, 09:15:18, 09:16:17), claimed at 09:16:59, and spent 85 s in `task.prepare`. Of that, `heartbeat.prepare.run_output` took 58.7 s, because it re-read the run's whole log from object storage. A control run spent 23 s in `task.prepare`. Queued starts used the same preparation path, the same 8 object-I/O slots and the same database pool, and nothing gave a reattach priority. The first-attempt failures were `PRP command session.snapshot timed out` and `codex_history_read_failed`. The retry then failed with `runner_harness_state_mismatch: resumed_not_suspended`, `runner_remote_process_identity_unavailable` or `provider_turn_lost_on_restore`. **Expected behavior** Every live run is reached within about 30 s of the swap. Queued work starts after that, or after a bounded wait. **Steps to reproduce** 1. Run a two-server release swap with live remote runs and queued runs. 2. Observe `native.recovery.transition` "waiting" events with `remote_controller_lease_live`, then the claim, then a long `heartbeat.prepare.run_output` span. 3. The new tests in `reattach-phase.test.ts` and `heartbeat-native-early-hand-off.test.ts` model this. **Paperclip version or commit** `master` at |
||
|
|
21bf7ebcee |
feat(connectors): connect Tailscale networks to SSH environments (#20617)
## Thinking Path > - Paperclip manages work for AI agents. > - SSH environments run tasks on computers that users own. > - A Paperclip server needs network access before it can reach a private computer. > - A reusable Tailscale connection belongs in Connectors. > - Connected machines should become choices in the existing SSH environment form. > - This change connects a tailnet and resolves selected machines through a private network client. > - Agents retain their choice of harness and AI account. ## Linked Issues or Issue Description **Subsystem affected** Connectors, environment settings, SSH execution, company secret storage, and runtime packaging. **Problem or motivation** Entering a Tailscale address in the SSH form does not connect the Paperclip server to that tailnet. Requiring a separate device key for every environment also duplicates network setup. **Proposed solution** Add a Tailscale Connector with **Connect Tailscale Network**. Use brokered OAuth consent to enroll one private network client per connection. Let SSH environments select a connected network and stable machine identity. Preserve the enrolled node state through the company vault so a restart does not reuse the one-time token. **Alternatives considered** A separate Tailscale environment type would duplicate SSH behavior and mix network access with harness selection. This change uses the existing `ssh` driver. Related #15648 adds OAuth client credentials and a health check; this change uses user consent and supplies outbound task execution. Related #11743 concerns HTTPS brokering. **Roadmap alignment** This extends the existing Connected Apps, environment, and Personal & Shared AI Accounts capabilities. This PR is stacked on #21248, which provides SSH account selection independently of Tailscale. The hosted OAuth broker is a companion control-plane change. ## What Changed - Preserve Cloud reauthentication in the current browser tab. Explain canceled or failed consent in the connector and offer a retry of the same authorized saved connection. - Record Cloud enrollment starts initiated through the Tailscale connector in the company activity log. - Check install-preparation descendant liveness through the kernel process table in the Runner support test. Keep the direct child’s strict reaping assertion and reject any non-zombie descendant. - Align the application image stage contract with the Tailscale client stage after the upstream Runner-image refactor. Keep the exact stage list and check every application stage, including `tailscale-client`, for prohibited Runner compiler commands. - Group Tailscale consent, discovery, daemon lifecycle, pairing, and deletion under `server/src/services/connectors/tailscale/`, with their service tests. Update existing route, runtime, environment, vault, and company-deletion imports. Keep the existing contracts, authorization, audit, and transaction owners. - Handle early stdin closure in the Kubernetes real-shell test helper. Stop the input stream on `EPIPE`, preserve nonzero child exits, and reject a zero exit when input was lost. Add two real-process regression cases. Production transfer code and confinement checks are unchanged. - Read configured Tailscale executable paths when a client starts, after supported `.env` loading. Keep explicit command overrides available. - Discard pending pairing results after switching to Direct SSH or closing the form. Retain the existing company, connection, machine, and port checks. - Store network credentials with an immutable server-owned purpose from their first vault row. Exclude them from generic catalogs, bindings, environment projection, and value reads. Use a dedicated audited server consumer for pinned connector reads. Block unclassified legacy serialized node state at the same value boundary. Keep SSH private-key binding separate from connector credentials. - Gate network queries and setup on verified local-implicit or instance-admin access. Hide Tailscale catalog rows for other members, show an admin requirement on direct setup navigation, and avoid device discovery before management access is known. - Keep failed-client cleanup in the network manager. A failed request cannot disconnect a newer retry for the same identity; explicit connector disconnect and enrollment retirement still stop their networks. - Check for readable Ed25519, ECDSA, or RSA host public keys before pairing edits `authorized_keys`. Print a known-hosts entry for every available key, or fail before changing authorized keys if none are available. - Return self-hosted Cloud enrollment to the Tailscale connector. Save the unprefixed setup path and let the shared callback add the verified company prefix. Keep external and unrelated destinations rejected. - Hide pairing instructions after an SSH port change. Ignore late key-selection results for the old port. Operators prepare again for matching known-hosts output. - Retire connector references before deleting a company or its vault credential. Stop company daemons after the database commit. Keep failed secret retirement retryable and prevent late node-state writes from restoring deleted connections. Discard retired state without blocking behind the vault deletion lock. - Reload a mismatched credential before replacing a ready daemon. Recheck the active entry and manager generation after the read. An older request cannot stop a newer healthy socket or cross shutdown. - Pin connector credentials to the exact vault version and SHA-256 digest recorded by trusted enrollment and node-state writes. Ignore generic secret rotations. Advance both fields with a compare-and-set guard. Reject unpinned or tampered credentials before network startup. - Save an explicit Direct SSH selection and clear old Tailscale fields after config merging. Preserve the operator's entered direct host. - Route all Tailscale SSH runtime access through the connection service after SSH-key loading. Pin access to the environment's saved connector identity and recheck after device discovery. Remove standalone key-based daemon startup and reject raw enrollment keys or credential-only network configurations. Align the shared config type and SDK pins. - Correct two E2E assumptions exposed during connector qualification. Keep the two-window chat fixture stale until its own mutation. Validate the original failed run and its permitted write-revocation timestamp in the legacy recovery browser test. - Add the Tailscale Connector page, catalog entry, OAuth return flow, connected-network management, device discovery, and disconnect. Document all six routes in OpenAPI and the API reference, including their instance-admin requirement. - Add company-scoped connection storage in `0480_lazy_lethal_legion`, trusted credential version/digest fields in `0481_round_pet_avengers`, and server-owned secret purpose/backfill in `0482_aberrant_rumiko_fujikawa`. Generate all three migrations through the maintained command and retain the five newest snapshots. - Add machine pairing that generates a fresh SSH key per request, stores its private half in the company vault, and uses the generated public half in the install command. Do not trust editable saved metadata. Preserve existing authorized keys, including a file without a final newline. Keep SSH username, workspace, and trusted host keys explicit. - Use the existing SSH driver. Resolve the machine's current address from its stable device ID. Reject disconnected, removed, offline, and wrong-company selections. - Hide Network when Direct is the only option. Select a sole connected tailnet automatically and hide its selector. Remove the requested setup helper copy. Do not restrict the harness to Claude. - Run a private userspace Tailscale daemon with a separate socket. Persist its node state before execution. Restore it after restart, bound startup, serialize reconnect cleanup, and close it during server shutdown. - Guard OAuth completion with a durable attempt marker. Disconnect and newer authorization invalidate older callbacks. Close replaced daemons and remove cancelled enrollment secrets without changing newer connection state. - Preserve SSH host verification and tailnet-specific session identity. Honor the configured Tailscale executable path, including spaces and OpenSSH percent escaping. - Package checksum-pinned Tailscale clients and their license in the standard runtime image. Exercise the client build stage in PR CI and include it in image smoke checks. ## Verification - Merge status: all CI and image checks pass, and current-head Apex is 5/5 with no unresolved comments. GitHub rejected a direct merge because the active master ruleset requires code-owner approval for `.github/workflows/pr-trusted.yml`. The [CODEOWNERS workflow pattern](https://github.com/paperclipai/paperclip/blob/master/.github/CODEOWNERS#L3) applies; eligible maintainers already have review requests. Auto-merge is enabled with the squash method for the verified head. The PR remains open pending that required approval; no branch policy was bypassed. - Final revision: `fdb29b2cc366f4945b7f4c3e2b2d25070329a21b`, including master `999709b0a6dd416af335b5debded13e1de7fcb7a`. [Final CI verification passed](https://github.com/paperclipai/paperclip/actions/runs/43518584814/job/132213508320). All required checks pass, all applicable test/build gates are green, and the branch is mergeable against master. SSH prerequisite #21248 and Cloud broker paperclipai/paperclip-cloud#1314 are already merged. - [Apex reviewed this exact revision at 5/5](https://github.com/paperclipai/paperclip/pull/20617#issuecomment-6776629437). There are no unresolved review threads. The PR is classified as Critical impact. - Final image qualification passes for the [application Runner path](https://github.com/paperclipai/paperclip/actions/runs/43518584522/job/132206784267), [amd64 Runner image](https://github.com/paperclipai/paperclip/actions/runs/43518584544/job/132207296198), and [arm64 Runner image](https://github.com/paperclipai/paperclip/actions/runs/43518584544/job/132207296156). Typechecks, Runner support, native/Rust tests, provider tests, serialized-server suites, browser shards, and the remaining applicable CI checks pass. This is source/image CI qualification, not a production rollout. - The last master conflict affected only Docker stage declarations and their boundary test. Both the checksum-pinned Tailscale stage and upstream `runner-lockfile` guard are retained, including the upstream `deps` inheritance. The Tailscale client stage is byte-identical. All 180 cases across five Docker/image contract suites pass locally, along with server and UI typechecks. The maintained 543-input SDK check has no drift. Runtime and lock pins are unchanged. Secret/PII and whitespace scans pass. - Seven OAuth regression cases failed before the fixes. The current-tab test uses the real handoff helper, checks that the session is stored before reauthentication navigation, mounts the shared return page, resumes consent, and verifies handoff cleanup. Callback tests check trusted saved-connection context, fixed safe outcomes, failure audit metadata, denied membership and expired-state rejection, JSON API errors, and no provider text in browser returns. UI retries require a connection in the current company's catalog. All 80 OAuth route/service/UI/shared-handoff cases pass; server and UI source/test typechecks pass. - [Server shard 4/12 on the prior refresh](https://github.com/paperclipai/paperclip/actions/runs/43510727031/job/132180301436) passed 2,784 cases and skipped 21, then failed a reassignment fixture's `TRUNCATE companies CASCADE` cleanup with PostgreSQL `40P01`. Its automatic retry found the surviving `REA` prefix. The unchanged test passes all three cases locally. A controlled reader/cleanup overlap reproduces the same deadlock, but the exact competing CI query was not logged. Each case now has a distinct company prefix and identifier; the owned database is removed at suite end. The three test bodies and assertions are byte-identical. Eight local reassignment/lock-release cases pass, and the [final server shard 4/12 passes](https://github.com/paperclipai/paperclip/actions/runs/43518584814/job/132207159392). Production reassignment logic, deadlines, and retries are unchanged. - The initial master integration passed 330 cases across 17 suites. Coverage includes connector state, device discovery, network lifecycle, pairing, routes, SSH binding and exports, environment persistence, and connector UI. All three unpublished migrations were regenerated after master's `0479_short_darkstar`: `0480_lazy_lethal_legion`, `0481_round_pet_avengers`, and `0482_aberrant_rumiko_fujikawa`. Each SQL body is byte-identical to its earlier unpublished migration. Generated snapshots preserve existing master tables and add only the intended Tailscale table and secret-purpose column. Exactly five snapshots remain, 0478–0482. Numbering, safety, compatibility, and snapshot-drift checks pass. The later Docker refresh did not alter this migration sequence. - Earlier attempts remain part of the verification record. On `1c020a0`, Rust cache restoration and server shard 5/12 were interrupted by shutdown signals, Runner shard 2/3 lost communication with no retrievable log, and an amd64 provider download failed with npm `ECONNRESET`. Same-head retries recovered, without establishing the cause of the communication loss. Later superseded runs also had cancellations with incomplete logs. These interrupted attempts are not counted as passes. The final revision has fresh successful CI. - Local platform limits remain recorded. The Kubernetes suite on macOS passed 191 cases, failed 12 Linux `/proc/self/fd` transfer cases, and skipped four Linux-only cases. Unchanged source reproduced the same 12 failures. An initial run also lacked provider dependencies until the maintained preparation command installed them. Those local attempts are not green qualification. Final Linux provider CI passes. Local Docker qualification was unavailable; final image checks passed in CI. - Before PR separation, a live tailnet/SSH Claude Code task completed on Ubuntu ARM64 without provider login for the remote user. The selected Paperclip subscription ran it, wrote the expected file, posted completion, and marked the task done. File contents and private runtime cleanup were checked. This is prior live evidence; the final revision was not live-retested. Other providers have automated coverage only. Hosted registration for unrelated tailnets and production concurrency remain qualification limits below. - The Superagent security and supply-chain reports remain neutral and inconclusive. Earlier attempts to restart the failed security scan returned HTTP 404. Supply-chain intelligence lacked coverage for extracted pinned dependencies and the shared workflow reference. These reports do not establish successful security qualification. The required CI and review gates pass; this external coverage gap remains disclosed. ## Risks - The Superagent security and supply-chain reports are inconclusive. The security scan re-run request returned HTTP 404, and the supply-chain scanner lacks coverage for the extracted dependencies and shared workflow reference. These neutral results are not successful security qualification. - The server-owned secret purpose is internal database metadata, not an HTTP-writable field. The generic secret service owns enforcement so direct API reads and runtime projection share the same restriction. Previously copied node state remains valid outside Paperclip until the enrolled device is revoked in Tailscale; a local disconnect cannot stop an external copy. - Tailscale network configurations now require a connected Connector and selected machine. The unreleased key-only path is removed; any saved configuration from that path needs Connector setup. - The connector remains a pilot. The local test used an OAuth app registered in the consenting user's tailnet. Qualify hosted app registration for unrelated customer tailnets before general availability. - This implementation supports one serving process per instance. Concurrent replicas need exclusive ownership of the persisted network identity before production rollout. Preserve the database and company vault master key across restarts. - Each machine-pairing preparation creates a new vault secret. Unused generated keys follow the existing secret deletion lifecycle. A saved SSH key remains available as an explicit choice. - Apply additive migrations `0480_lazy_lethal_legion`, `0481_round_pet_avengers`, and `0482_aberrant_rumiko_fujikawa` before application rollout. The new credential version and digest columns are nullable. Existing unpinned connections require fresh OAuth consent; the migration deliberately does not trust or backfill a generic secret value. Regenerate or renumber migrations if master advances the sequence before merge. - The target needs an SSH server, its chosen harness, and allowed tailnet access. A reachable network does not grant SSH or provider authorization. Pairing reads standard `/etc/ssh` host-public-key files; custom host-key locations require manual known-hosts configuration. - Roll back the application revision to remove new entry points. Existing connection secrets and registered Tailscale nodes retain their separate lifecycle. Do not drop the table or delete node state as part of an application rollback. - Production qualification still needs denied-policy, revocation, cancellation, and deployment-overlap checks. Local Docker and a production deployment were not performed. ## Model Used OpenAI Codex, gpt-6.1-sol, xhigh reasoning, with local code execution and tool use. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used - [x] I have checked ROADMAP.md and this extends existing connector and environment features - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR using the feature request fields - [x] I have not referenced internal issues, local URLs, or private machine identifiers - [x] My branch name describes the change - [x] I have run focused tests locally and they pass - [x] I have added tests for new behavior and failure paths - [x] I have updated documentation - [x] I have documented migration and runtime risks - [x] All Paperclip CI gates are green - [ ] Required code-owner approval received for the workflow change - [x] Greptile Apex is 5/5 on the final head with no unresolved comments - [x] I will address CI and review findings before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
944b1c64d7 |
feat(mcp): keep long provider calls durable with the MCP Tasks extension (#21263)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents use connected apps through Paperclip's MCP client in the tool gateway > - Some provider actions run for minutes: CI runs, deploys, batch jobs > - Today such a call is lost if it outlives the tool timeout or if Paperclip restarts or deploys mid-call, and a retry can start the work a second time > - Recovery must just work, and the target rate of runs that need a manual retry is 0% > - This pull request implements the MCP Tasks extension (`io.modelcontextprotocol/tasks`, 2026-07-28) in the MCP client > - The benefit is that a long provider call becomes a durable task: it survives timeouts, restarts, and deploys, and it is never started twice ## Linked Issues or Issue Description **Problem** A long provider tool call holds an HTTP request open. When it passes the tool timeout, or when Paperclip restarts or deploys during the call, the result is lost. The agent then often repeats the call, and the provider does the work again. **Proposed solution** Implement the client side of the MCP Tasks extension. Declare it on stateless tool calls. Store each returned task durably, wait on it during the call, keep polling it in the background after the call ends, and let the same call attach to the stored task instead of sending `tools/call` again. **Why it matters** CI, deploy, and batch actions are where agents lose the most work today. A durable task handle makes these actions safe across timeouts, restarts, and deploys, without a person. Refs #21119, Refs #21140, Refs #21219 (the MCP Apps, stateless MCP client, and MCP Skills work this builds on). ## What Changed - **Off by default:** MCP Tasks are behind the new instance experimental setting `enableMcpTasks` ("Durable app actions (MCP Tasks)"), default off. When it is off, Paperclip never declares the extension, nothing attaches to or polls tasks, pending rows do not block idle sleep, and behavior matches master. A test covers the off state. - `mcp-http.ts`: the tasks extension is declared per request, only on HTTP `tools/call` and `tasks/*`. Discovery, listings, and stdio servers do not declare it. `tasks/*` requests put the task ID in `Mcp-Name`. - New `mcp-tasks.ts`: parses task results, bounds the server's timing hints, sends `tasks/get`, `tasks/update`, and `tasks/cancel`, and seals task IDs at rest (AES-256-GCM, bound to company and connection; a keyed hash gives uniqueness). - New company-scoped table `tool_mcp_tasks` (migration `0475`, online-compatible). It links the task to the call, run, task, connection, and grant. It stores no provider result. - `tool-gateway.ts`: - A returned task is recorded, then polled up to the normal tool timeout. Its result goes through the same validation, redaction, and recording as a direct result. - Past the timeout, or when the caller disconnects, the agent gets `provider_task_pending` ("call again with the same arguments"). The task stays durable. - A call with the same endpoint, tool, full arguments, grant, agent, task, and actor attaches to the undelivered task. A retry with the same idempotency key also attaches, and it can never start different work under that key. - Input: URL requests show the provider link (Paperclip never answers them). Form requests become a question card; only the person's answer is sent with `tasks/update`. Other requests stop with a clear "needs input" error. - Stopping the run sends `tasks/cancel` and marks the task cancelled. A caller that only disconnects keeps the task. - `failed` becomes a tool error with the provider's message. An unknown task or a passed TTL becomes `provider_task_expired`. Every task stops being tracked at its TTL, or after 7 days at most. - Audit entries: `tool_gateway.task_started`, `task_input_sent`, `task_cancelled`. - New `mcp-task-work.ts`, registered at startup with the shared deadline scheduler: it resumes polling pending tasks after a restart, cancels tasks whose task closed or run was stopped, and wakes the assignee once when a task settles after its run ended. - Docs: `doc/connections/MCP-TASKS.md`, linked from `doc/connections/README.md`. Background work (per `rules/sleep-aware-background-work.mdc`): trigger is a task row write or release (`mcp-tasks` queue) and one startup pass. The deadline is the earliest `next_poll_at`. Cadence is the server's `pollIntervalMs` within 5 s – 5 min, with backoff; 5 min while waiting for a person. It is quiet when no row has a next poll. This is the narrow external-source exception: providers give no wake Paperclip can receive, and polling stops with the task. ## Verification - `server/src/__tests__/mcp-tasks.test.ts`: 20 tests with a fake 2026-07-28 server that returns tasks only to a client that declares the extension and checks `Mcp-Name`. They cover: - polling to completion, with the result identical to a synchronous call; - timeout, then an attaching call with no second `tools/call`; - a keyed retry, and a reused key with different arguments (refused); - restart mid-task: a fresh gateway with no in-memory state polls the row, wakes the agent once, and the recovered run gets the result once with no second `tools/call`; - no wake while the run is still going; - URL input (link, never answered), and form input (question card, then `tasks/update` with the answer); - cancellation by the run, by closing the task, and no cancellation when only the caller leaves; - failure, an unknown task, and TTL expiry (ends in under 3 s, no hang); - another agent never attaches; - synchronous calls unchanged and older servers never see the extension. - Red/green: with attach disabled, the timeout, keyed-retry, and restart tests fail (3). With `tasks/cancel` disabled, both cancellation tests fail (2). Before `tools/call` declared the extension, 13 end-to-end tests failed. All pass with the change. - Existing suites pass: `mcp-protocol-versions`, `mcp-http`, `mcp-apps`, `mcp-skills`, `server-startup-feedback-export`, `tool-gateway`, `tool-gateway-discovery-http`, `remote-mcp-connectors` (342 tests). - `tsc --noEmit -p server` is clean. `pnpm --filter @paperclipai/db check:migrations` passes (the migration is classified online-compatible). Snapshots pruned to five. - Agent boot and time-to-first-token: no change in the boot or first-turn path. The only new per-call cost is one indexed lookup on stateless tool calls. - Not run: a live provider that returns tasks. No public MCP server that returns tasks was available without new credentials. ## Risks - Shipped off by default. Review findings that only affect the setting-on path after this merge are tracked as follow-ups. - Migration: a new table with indexes and foreign keys to `companies`, `tool_invocations`, and `tool_connections`. Online-compatible. - Behaviour change only for servers that return tasks, which they may do only to clients that declare the extension. Older servers and stdio servers are unchanged. - A task that settles while nobody is waiting stays attachable for 1 hour, so a deliberate repeat of the same call within that hour collects the earlier result instead of starting new work. - The wake after a background task settles uses the generic task wake; the agent must call the tool again to collect the result. - Follow-ups: `notifications/tasks` via `subscriptions/listen`; sampling and roots input requests; showing a long action's progress in the task chat. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits, local test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
500bc694b5 |
fix(recovery): keep a draining server's lease alive and end a crashed server's runs in about 2 minutes (#21238)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A release swap runs two servers for a short overlap. The old server drains, and the new server takes over the runs. > - Each server proves that it is alive with a controller-instance lease. Recovery uses that lease to decide whether a run's controller is gone. > - In local swap cells, a healthy old server exited during the overlap, because its lease renewal waited for a connection on the busy main pool. Every run it drove was interrupted. > - When a controller crashed, its runs waited for another server boot, or for a timer that restarted on every look. Tasks stood still for 21 to 28 minutes. > - This pull request renews the lease on its own connection, and lets the live server end a crashed controller's runs within a bounded time. > - The benefit is that a swap does not kill a healthy server, and a crashed server's runs end in about 2 minutes, so their tasks continue by themselves. ## Linked Issues or Issue Description Refs #21135, #21222 **What happened?** 1. During a two-server release swap (local cells, load 14 to 34), the outgoing server exited with "controller instance lease lost while holding work". Five lease renewals in a row timed out after 10 s. The 10 s timeout included the wait for a connection on the shared main pool, which the traffic switch and the replacement's startup pass kept busy. 2. When the old server was killed in the middle of the overlap, its runs stayed in progress for 21 to 28 minutes. The live server did not re-check a crashed controller's runs: only a new boot did. The unverified-controller wait started when a server first looked at the run, not when the controller left, and the successor's own wait record counted as the dead owner's sign of life. **Expected behavior** A healthy server keeps its lease during a swap. When a controller is gone (its lease lapsed and it is fenced or stopped), the live server ends that controller's runs within a bounded time, without another server boot, so restart recovery continues the tasks. **Steps to reproduce** 1. Run two servers against one database with a release swap and a 200 s overlap. 2. Kill the old server 40 s into the overlap (SIGKILL). 3. See that its runs stay in progress for more than 20 minutes, until another server starts. ## What Changed - Lease renewal on its own connection (`packages/db/src/client.ts`, `packages/db/src/index.ts`, `server/src/services/controller-instance-lease.ts`): - New `controllerInstanceLeaseDb(db)`: one private 1-connection pool (`application_name` `paperclip-controller-instance-lease`), made the same way as the existing `controllerLeaseDb` and closed by `closeLivenessDb`. - The renewal is one statement on that connection, as before. - The connection sends `statement_timeout` = 8 s (below the 10 s client timeout) as a startup parameter (new `DatabaseClientOptions.statementTimeoutMs`, used only by this pool). The server cancels a slow renewal, so it never holds the single connection that the next renewal needs, and there is no extra round trip per renewal. (An earlier revision set it per transaction; that added three round trips to every renewal and broke the idle-server receipt test.) - If a connection pooler refuses the startup parameter ("unsupported startup parameter"), the lease logs a warning and renews on a replacement connection without it. - A crashed controller's runs (`server/src/services/recovery/ownership-registry.ts`, `server/src/index.ts`, `server/src/services/native-runtime/native-restart-recovery.ts`): - The ownership reconciler on the live server hands runs whose controller is proven gone to restart recovery. It does not wait for another server boot. - The unverified-controller wait is bounded from the moment the controller's lease lapsed, not from the first look. - When the controller boot is `stopped` or `fenced`, the run ends at once. - A successor's wait record no longer refreshes the dead owner's sign of life. Safety of ending a run when its controller boot is fenced or stopped: - Only `stopped` (written by the boot's own graceful stop) and `fenced` (a newer epoch fenced it after its lease expired plus 30 s grace) count. `live` and `grace` never count, so a controller that is alive and renewing is never treated as gone. `expired` alone does not count either. - A boot that cannot renew stops itself at its self-fence window (about its lease expiry), and a boot that finds itself fenced exits at once. So when another server can fence it, the old process has already stopped driving runs. The lease change above removes the main cause of healthy boots failing to renew. - If the old process is still alive (partition or stall) and its Runner continues: the old run is fenced. Its agent token is refused for writes (`agent_run_superseded`), its remote workspace epoch is abandoned or the continuation gets a new workspace, and the native coordinator records `replacementDenied`. There are never two writers in one workspace. ## Verification Red-to-green tests (each failed before its fix and passes after it): - `server/src/__tests__/controller-instance-lease.test.ts` "own connection": red (10004 ms timeout while the main pool is held) to green. Full file green (26). - `server/src/__tests__/controller-instance-lease.test.ts` "has the server cancel a renewal blocked on a row lock, and renews on the same connection next time": red without the connection-level timeout (the renewal runs to the 10 s client timeout and stays active), green with it (cancelled at about 8 s, no active lease statement left, the next renewal succeeds in under 2 s on the same backend). - `packages/db/src/client-options.test.ts`: the statement timeout is sent as a startup parameter. - `server/src/__tests__/heartbeat-native-restart-suspension.test.ts`: "lapsed longer ago" / "end of the bound" (wait counted from the lease lapse), the live-server re-check of a crashed controller, the fenced/stopped boot ending the run at once, and the successor's wait record. Full file green. Changed existing expectation (it asserted the old wait for a stopped or fenced controller): - `server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts` "keeps foreign local Runner and provider ownership after its controller is stopped/fenced": the disposition is now `blocked` (`controller_process_space_unverified`) instead of `awaiting_evidence`, the run is `failed` with `native_restart_recovery_blocked`, and the finalization is `terminal_failure` with `executorStopEvidence: "unverified"`. The safety checks stay: no local PID probe, no claim, no stamp, no local stop proof, the run keeps its process identity and session checkpoint, and no result is written. Local commands, on this branch based on current master: - `cd server && npx vitest run src/__tests__/controller-instance-lease.test.ts src/__tests__/heartbeat-native-restart-suspension.test.ts`: 2 files, 140 tests passed. - `npx vitest run src/__tests__/controller-instance-lease.test.ts src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`: 225 passed, 14 skipped (real-process cases that need CI). - `pnpm exec vitest run src/__tests__/maintenance-server-routes.test.ts -t "releases an ordinary held task Retry"`: passed (idle receipt: one lease statement after the work ends). Local swap cells (two servers, embedded Postgres, real agents, old server SIGKILLed 40 s into a 200 s overlap): - Before: runs of the killed server stayed in progress 21 to 28 minutes. - N1 (2 tasks): 2/2 finished hands-off. Kill to run end 5 min 31 s (the successor's wait write still moved the timer; fixed by the last commit). - R1 (6 tasks, 3 adapters, subtasks, follow-ups, 2 swaps): 4/4 hands-off, 0 banners, 0 duplicates, 0 invariant violations. Kill to run end 3 min 39 s. - N2 (same seed as N1, all fixes): 2/2 hands-off. Kill to run end 2 min 01 s; fresh runs started at 2 min 32 s and 3 min 02 s. The floor is the lease TTL (90 s) plus the fence grace (30 s). - Hour-long release-swap wave (w-20261112T233630-s7201) on the branch that holds this fix and the related recovery fixes: 11/11 tasks finished hands-off (master: 9/11). The lease and crashed-controller fixes were visibly working in it. ## Risks - A pooled database URL whose pooler refuses `statement_timeout` as a startup parameter: the first renewal fails over to a connection without it (logged), so the lease keeps working without the server-side cancel. - One extra database connection per server, held while the server runs. It is idle except for one renewal every few seconds. - A run whose controller boot is fenced or stopped now ends at once instead of after the bound. See the safety notes above: only a boot that is not alive can be in these states. - No migration. No schema change. Nothing changes before `turn.start` on the normal path. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use (shell, file edit, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes (no documented behavior or command changes) - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
2de5d97958 |
fix(startup): report ready without waiting for backlog first passes (#21247)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - A hosted instance upgrades by starting a new server beside the old
server. The deployer sends traffic to the new server when `GET
/api/ready` returns 200.
> - `/api/ready` returns 200 only after the boot sequence in
`server/src/index.ts` sets the phase to "ready".
> - The boot sequence awaited the first pass of many delivery queues.
Each pass grows with saved backlog or waits on outside services.
> - On a busy instance, these passes took more than the deploy's
health-check window. The deployer killed two new servers in a row before
they became ready, including a rollback to an image that had started in
86 s one hour before.
> - This pull request moves those first passes off the readiness path.
Readiness now waits only on named, bounded preconditions, and each step
reports its duration.
> - The benefit: an upgrade or rollback becomes ready in boot time, not
in backlog time. A slow start also names its slow step in one log line.
## Linked Issues or Issue Description
**What happened?**
On the hosted staging instance, two replacement servers did not report
`/api/ready` and were killed after about 9 minutes. The first was a new
image and the second was a rollback to the old image. At that time the
instance had about 200 queued runs, about 80 running runs, a slow
database, and many continuation runs. The failed server logged "managed
environments ensured from managed config" (`index.ts` line 1284 on
master). It never logged "worktree run-execution cutoff state" (line
1726), which the server always logs when the heartbeat service exists.
Between those two lines, boot awaited only these steps: `tool_reviews`
(four tool-review queue first passes), `gateway_sessions`,
`delivery_workers` (budget enforcement, cost accounting, fast responses,
chat completions, connection continuations, tool action receipts and
question responses), the GitHub connection event and continuity workers,
and `terminalWorkspaceWorker.ready`. Each step is a delivery-coordinator
`register(...).ready`, and that promise resolves only after the queue's
whole first pass. No queue except `mcp-client-mirror` (also failing on
the healthy server) logged an error, so the cause was slow passes and
not failures. The log had no per-step markers, so it does not show which
of these awaits was still running at the kill.
**Expected behavior**
Readiness means "this server can serve requests and take over". It must
not wait on work that grows with backlog or that depends on an outside
service. Queues own that work and retry it on their own schedule after
readiness.
**Steps to reproduce**
1. Start the server while the first pass of any of the queues above does
not finish. The new test in `server-startup-feedback-export.test.ts`
does this for every queue.
2. On master, `startServer()` does not resolve and the phase stays
"recovering".
## What Changed
- `server/src/index.ts`: These first passes now start in the background
as tracked scheduler work and do not hold readiness: tool reviews,
gateway sessions, delivery workers, GitHub connection workers, the
terminal workspace sweep, environment lease cleanup (scheduler-disabled
path) and scheduled database backups. Shutdown and idle sleep still wait
for them through `trackHeartbeatSchedulerWork`, and each queue keeps its
retry schedule.
- `server/src/index.ts`: The restart backup after an idle wake (a
`pg_dump` whose time grows with the database) holds idle eligibility
(`markIdleStartupComplete`), as before. It no longer holds readiness.
- `server/src/index.ts`: Every remaining await on the boot path is a
named `startupTimeline.precondition(...)`: `listen`,
`runtime_control_operations`, `execution_policy`,
`managed_environments`, `worktree_run_execution`,
`scheduling_suppression`, `native_recovery`, `external_adapters` and
`cloud_configuration`. These preconditions do not change.
- `server/src/startup-timeline.ts`: Adds `precondition(step, work)` and
`background(step, work)`. A precondition that is still running after 30
s is named in the warning "startup step is still holding readiness",
which repeats every 30 s. A background step logs a warning once at 30 s
and logs its duration when it finishes late. A background step that
fails is logged with its step name. The line "startup timeline at
readiness" now includes `readinessStepsMs`, `backgroundStepsMs` and
`backgroundStepsOutstanding`. All timers are single unref'd timeouts and
the module adds no queries.
- `server/src/__tests__/startup-readiness-boundary.test.ts`: This guard
test parses `index.ts` with `parseAst` from `vite`. The server already
depends on `vite`, so this PR adds no dependency and no lockfile change.
The test fails when an await on the boot path (between the phases
"recovering" and "ready") is not a named precondition, or when the
precondition names do not match a reviewed list that gives the reason
for each name.
- `server/src/services/terminal-workspace-work.ts`: Updates a comment.
The first sweep no longer holds readiness, and it still skips
compaction.
## Verification
- Red/green regression test: `server-startup-feedback-export.test.ts`,
"reports ready while backlog queues' first passes never finish". This
test makes every first pass above never finish. With master `index.ts`
it fails with `expected 'still starting' to be 'ready'` for both
scheduler settings. With this change it passes.
- Guard test: with master `index.ts`,
`startup-readiness-boundary.test.ts` fails and lists 18 unnamed awaits,
including `timeStartupRecovery("delivery_workers")` and
`terminalWorkspaceWorker.ready`. With this change it passes.
- `cd server && npx vitest run
src/__tests__/server-startup-feedback-export.test.ts
src/startup-timeline.test.ts
src/__tests__/startup-readiness-boundary.test.ts
src/startup-recovery-timing.test.ts
src/__tests__/terminal-workspace-work.test.ts`: all tests pass.
- `cd server && pnpm typecheck`: passes.
- `pnpm verify` exited with code 2 on the first revision (a lockfile
change that this PR no longer makes). Required CI supplies the broad
coverage.
- After deploy, run `grep "startup timeline at readiness"` on a server
start. The line shows each precondition's duration and the background
passes that are still running.
## Risks
- The listed first passes now run at the same time as native run
recovery and the requests that the server serves, not before them.
Requests were already served during recovery. Commit-woken passes of the
same queues, and the fresh queued-run dispatch, already ran at the same
time as native recovery. The coordinator still serializes the passes of
each queue.
- `/api/ready` can now return 200 while a queue still has backlog. That
is the intended meaning, and it matches the queued-run and recovery
passes that already run after readiness (#21014).
- Native run recovery still holds readiness, which is correct because
adoption must happen before takeover. If that recovery is slow, the
server now logs its step name every 30 s.
- Low risk otherwise: no schema or API change.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude (Anthropic) via Claude Code, model ID `claude-opus-5-5` (commit
trailer: Claude Fable 5.2). Tool use and code execution were enabled.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
|
||
|
|
dfc510b887 |
fix(server): hand new run starts to the replacement server during a deploy overlap (#21127)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Paperclip Cloud replaces the server process on each release. The old server keeps serving until the new one passes health checks and takes the traffic. > - During that overlap the old server keeps starting new runs right up to SIGTERM, although the new server can already start them. > - SIGTERM then cuts off exactly those runs while they still prepare. In one audited swap the old server started 30 runs after the new server booted, and 23 of them ended `process_lost`. > - Admission must never pause, so the fix must hand over who starts new runs, not stop admission. > - This pull request makes the new server record when it starts runs. The old server then leaves new run starts to it, and starts them again if the new server stops or dies. > - The benefit is that a deploy no longer starts runs on a server that is about to stop, and the company always has one server that starts runs. ## Linked Issues or Issue Description Refs #20852 (preserves preparing and launching turns that a deploy interrupts; this PR reduces how many such turns the old server starts). Related: #21046, #21107 (shared controller-instance owner proof), #20986 (queued-run dispatch signal). **What existing behavior does this improve?** The start of new runs while two servers overlap during a Cloud deploy. **Subsystem affected** Server run admission (`server/src/services/heartbeat/queue.ts`), the controller instance table, and startup and shutdown in `server/src/index.ts`. **Current behavior** The old server starts queued runs until it receives SIGTERM. Runs that it starts in the last minutes of the overlap are interrupted while they prepare. Nothing tells the old server that its replacement already starts runs: `controller_instances.epoch` is written at process start, and that happens minutes before the new server can start runs. **Proposed behavior** The new server records `dispatching_at` and a short `dispatch_lease_expires_at` on its `controller_instances` row right after its startup queued-run pass. The old server reads this through an in-memory flag at run admission and leaves new run starts to the new server. Runs it already started continue and are handed over at SIGTERM as today. If the new server stops, drains or dies, the old server starts runs again within seconds. **Reason and benefit** Fewer runs are cut off by each swap. There is never a moment without a server that starts runs. **Breaking changes** None. The migration adds two nullable columns. Releases without this code ignore the columns, and their rows never carry the marker, so a mixed-version overlap behaves as today. ## What Changed - `controller_instances`: new nullable columns `dispatching_at` and `dispatch_lease_expires_at` (migration `0471_controller_instances_dispatch_marker`, two `ADD COLUMN` statements, no lock beyond the catalog update). - New `server/src/services/run-start-handover.ts`. It is one worker on the shared delivery coordinator (`DELIVERY_QUEUES.runStartHandover`). Its renewal and recheck are coordinator deadlines, and a drain change or expiry wakes it. - Replacement side: `begin()` records the intent and writes the marker (it creates the boot's row if the row is missing). If the server drains at that moment, the worker writes the marker when the drain ends. The worker then publishes a `run-start-handover` hint and renews the 20 s lease every 5 s. It renews only while an older boot that is not stopped or fenced may still serve, and for at most 30 min. All marker writes run in one serialized chain. `release()` at shutdown lands after any renewal already in flight. A clear that fails is retried, and shutdown waits for the release. While the server cannot dispatch (drain, quiesce, admission closed), the worker clears the lease. - Old-server side: `isYielding()` is an in-memory read. The hint refreshes it, and so does one read that a dispatch pass waits for (at most 1 s) when the last read is more than 2 s old. A read that gets no answer within 1 s never keeps the server yielding. The server admits, and the yield also ends when the replacement's last seen lease runs out on the old server's own clock. Reads are a single flight on the dedicated connection. While it yields, the server rechecks every 3 s, calls `onResume` when the marker is gone, and sends a coalesced queued-run-dispatch hint for skipped starts. - It yields only to a newer epoch with a live marker. It never yields to an older boot, to itself, or to a row without the marker, and it does not yield without a row of its own. - **Database invariant:** every statement of the hand-over runs in a small transaction on the dedicated one-connection controller-lease pool (`controllerLeaseDb`) with a 1 s statement timeout. That covers reads, marker writes, the `run-start-handover` notification (commit-bound, in the marker write's own transaction) and the skipped-start hint. None of it touches the shared pool or registers tracked database work. The pool is shared only with legacy controller-lease renewals, and never with the liveness pool that native lease renewals use. A hand-over statement waits at most one legacy batch (10 s), inside the 15 s between a marker renewal and the marker's expiry. It delays a legacy renewal (60 s lease, renewed every 10 s) by at most 1 s. I did not piggyback on the controller-instance lease renewal, because that renewal runs only while the server holds work and only every 30 s. - **Import audit of `run-start-handover.ts`:** - `drizzle-orm` `sql`: builds statements, no I/O. - `controllerLeaseDb`: the dedicated connection, used by every statement here. - `notifyDatabaseWorkOnCommit` (new in `work-signals.ts`): runs `pg_notify` only on the transaction it is given and registers no intent, so the coordinator's `reconcileDatabaseWork` has nothing of ours to check. - `logger`: no I/O to the database. - `controller-process-identity`: in-memory boot id and host hash. - `delivery-work-coordinator`: type only. The worker is registered for lifecycle and deadlines; its `run` awaits only the serialized marker chain, and reads are not awaited. - `DELIVERY_QUEUES`: constants. - `task-admission` drain subscriptions: in-memory events. - `heartbeat.ts`: run admission is also closed while the server yields. A `runStartHandover` service option (default: the process's registered hand-over) lets one process host two assembled services in tests. `queue.ts`: `startNextQueuedRunForAgent` and `resumeQueuedRunsAfterDrain` first call `refreshRunAdmission()`. That call returns `null` (nothing to await) unless the hand-over read is stale. - `index.ts`: creates the hand-over only when the instance lease is on, the deployment notification channel exists, and the scheduler is enabled. It calls `begin()` after the RunDispatch startup pass and wakes that queue. It releases the hand-over when run admission closes at shutdown, and stops it before the lease stops. - `deployment-notifications.ts`: keeps the LISTEN connection open while a hand-over involves this server, so skipped starts and lease changes reach the other server at once. - `doc/execution-semantics.md`: new section "Run starts during a deploy overlap". It covers the marker, who yields, failure handling, what counts as starting a new run, and what a post-deploy audit should show. **Marker contract for other components (for example the deployer):** a replacement starts runs when its `controller_instances` row has an `epoch` greater than the old boot's, and `dispatching_at is not null and dispatch_lease_expires_at > now() and stopped_at is null and fenced_at is null`. `dispatching_at` stays set after the lease lapses, as the time the boot began starting runs. ## Verification - `cd server && npx vitest run src/__tests__/heartbeat-run-start-handover.test.ts`: assembled-app test with two `heartbeatService`s, each with its own pool, listener, coordinator and boot. A wake that the old server accepts stays queued there. The replacement's real queued-run dispatch starts it, and claims are observed per pool. After a replacement release, the old server starts the run itself. Red: with the yield removed from `isRunAdmissionClosed`, the old server claims the run. - `cd server && npx vitest run src/__tests__/run-start-handover.test.ts`: 19 tests pass against embedded PostgreSQL. Each test uses two servers, each with its own pool, boot id, LISTEN connection and delivery coordinator. The tests cover: - the old server is ready and the new server starts; the new server sets the marker; the old server yields; the queued run is claimed only by the new server, and a skipped start wakes the new server's dispatch - the old server learns from its own dispatch path without a notification: a read older than 2 s is refreshed before the start - the new server dies after it sets the marker: the old server resumes and runs its queued-run pass - the new server shuts down: the old server resumes at once - the new server drains: it releases the marker and takes it back afterwards - a replacement that starts up during a drain takes over when the drain ends - a shutdown release during an in-flight renewal stays released - a clear that fails at shutdown (statement timeout behind a row lock) is retried by a later release - an older server that outlives the lease TTL keeps yielding while the replacement renews - a yield that the old server cannot confirm within 1 s (its only connection is busy) does not stop starts (red/green) - a yield ends when the replacement's lease runs out, even while every read waits for a connection (red/green) - renewals blocked past the TTL hand starts back once, then the replacement takes them again (bounded flapping) - a clear that fails when the replacement starts draining is retried well inside the lease (red/green) - with both servers' shared pools blocked past the TTL, the marker is written, renewed and released, and the old server yields and resumes, all before the block ends (red when hand-over statements use the shared pool) - an older-release row without the marker changes nothing - a server never yields to an older boot or to itself, and does not yield without a row of its own - a lone server writes once, never renews, and admits from memory - without a hand-over (self-hosted) there is nothing to await - Red: on `origin/master` the test file cannot run, because the module and the columns do not exist. The old server has no signal and keeps starting the handed-over run. - `cd server && npx vitest run src/__tests__/deployment-notifications.test.ts src/services/heartbeat/dispatch-work.test.ts src/__tests__/delivery-work-coordinator.test.ts src/__tests__/controller-instance-lease.test.ts src/shutdown.test.ts`: all pass. - `cd packages/db && pnpm check:migrations`: safety and compatibility checks pass. `cd server && npx tsc --noEmit -p .`: clean. - `pnpm verify` asks for broad verification because `doc/execution-semantics.md` is a global CI input. Required CI supplies that coverage. ## Risks - Run starts move from the old server to the new server during the overlap. If the new server cannot dispatch, the old server resumes on the lease clear (about 3 s) or the lease lapse (up to about 23 s after a crash). If a hand-over read fails, the server keeps admitting. - If the old server was not listening for notifications, it can start runs for up to about 2 s after the marker (the age of its last read). On a dispatching server, the pre-dispatch read adds one small bounded query at most every 2 s. - An older boot that has been idle for more than 30 min counts as gone. The new server then does not renew, and the old server admits again as today. - The migration adds nullable columns only. The deployer can apply it while the previous release serves. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits) and extended thinking. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
b853cc28ae |
perf(server): reach readiness without waiting for backlog-sized recovery passes (#21014)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The server process starts, recovers saved work, and then reports ready on `/api/ready`. The deploy health gate waits for that signal. > - On a busy staging workspace, one boot took 85 s to bind the listener and 4.4 minutes to report ready. Small, idle stacks take 20–90 s. > - Two parts of that time do not need to block readiness. The server published several hundred UI assets one at a time before it bound the listener. After the listener bound, it waited for the first pass of each periodic recovery queue, and those passes grow with the backlog. > - The volume-free data gate, which can refuse a start, ran last. By then the server had already dispatched queued runs that a refusal would abandon. > - This pull request moves the refusal gate to the start, publishes the assets 8 at a time with replayable bodies, and runs the backlog-sized first passes after readiness, in their original order. > - The benefit is a faster time from container start to ready, and no work admitted before a gate that can refuse to start. ## Linked Issues or Issue Description No public issue exists. This is the problem description (enhancement template). **Is your enhancement related to a problem? Please describe.** Server boot time grows with the saved backlog. These timestamps come from the log of one staging boot (release |
||
|
|
8818402e7a |
feat(inbox): route requests to people and record personal outcomes (#20959)
## Thinking Path > - Paperclip helps people manage AI agents and their work. > - The inbox collects requests that need a human decision. > - Teams need to know who owns each request and who may act on it. > - A shared close operation must differ from hiding a card for one person. > - This pull request adds personal routing and durable outcomes to the existing attention service. > - People can claim or transfer work and inspect what another person did for them. > - Domain permissions still decide who may resolve each request. ## Linked Issues or Issue Description **Subsystem affected** Server, shared contracts, and database. This is the routing foundation for a stacked inbox UI change. **Problem or motivation** A company-wide attention feed does not give teams a clear personal queue. Archive, withdrawal, transfer, and decisions by another person need distinct scopes and records. **Proposed solution** Resolve a request audience from its addressee, task participants, and configured owners. Provide personal, unassigned, company, all-assignee, and archived views. Add claim, transfer, personal archive and snooze, shared withdrawal, receipts, and personal history. **Alternatives considered** A single shared inbox leaves ownership and personal hiding ambiguous. The service keeps one canonical request and projects its audience instead of copying requests for each person. **Roadmap alignment** This extends the existing Multiple Human Users and Agent Reviews and Approvals capabilities in ROADMAP.md. **Additional context** Refs #19363 and #19520 for the current inbox navigation and confirmation replacement rules. Refs #20268 for related actionable-request work. That PR filters actionability; this stack adds audience routing and personal outcomes. This is part 1 of a two-PR stack. The inbox UI PR #20961 targets this branch. ## What Changed - Add nullable company and project decision owners and a company budget owner. - Add audience views and personal badge counts to the canonical attention projections. - Add claim, transfer, withdrawal, personal archive, snooze, and restore operations with permission checks and activity records. Lock addressed transfers to the current addressee or routing administrators, and validate the recipient against canonical resolution policies. - Add receipts and scoped history for work handled by another person. - Keep closed-task requests and replaced confirmations out of current cards. - Add an idempotent migration and a concurrent receipt-history index. - Update typed fixtures so the existing UI remains compatible with the new contracts. ## Verification - Server layered typechecking passes, including all three test-tree partitions. - Seven routing and interaction-resolution suites pass all 249 tests against a dedicated external PostgreSQL test database on `4caecc21d8`. Six new regressions failed before the repair; they cover takeover followed by approval, legitimate delegation, excluded creators/review requesters, and unauthorized claims. - Prior focused verification covers private approval history and receipts, transfer visibility, audience escalation, withdrawal governance, hidden-card counts, per-recipient confirmation replacement, and SDK source contracts. - A synthetic three-person company passed 37 routing and lifecycle acceptance checks. The UI stack adds browser acceptance for the shared controls. - The complete [GitHub Actions CI matrix](https://github.com/paperclipai/paperclip/actions/runs/43134997331) passes on `4caecc21d891677323a7335f23b1a015cce24c13` (Linux/Node 24: typecheck, build, general and serialized tests, Runner support, browser shards, and aggregate gates). Greptile is 5/5 on that exact head with no open findings. The fresh security scan passes after explicitly rechecking the addressee-transfer repair. - `pnpm verify` reports exit 2 because a CI-global script changed; full CI supplies that broader coverage. Embedded PostgreSQL could not start on this Mac because shared-memory slots were exhausted, so local database checks use a separate external test database. Linux CI covers the embedded setup. ## Risks - Personal badge counts change to the viewer's assigned requests. Accessible feed totals and task membership keep their existing scope. - The migration adds nullable owner fields and a dismissal fingerprint. Its index runs outside a transaction. The migration is classified and rerunnable. - Routing changes the addressee used by resolution authorization. The current addressee or a routing administrator must authorize an addressed transfer; the recipient must satisfy writer access, canonical resolver exclusions, and task review policy under row locks. - The shared validator export barrel crosses 1,000 lines because it exports the new attention schemas. Schema implementation stays in the dedicated attention validator module. ## Model Used OpenAI GPT-6 through Codex. Used repository tools, code execution, and browser checks. This session does not expose a more specific runtime model ID or its context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal Paperclip ticket id - [x] I have run focused tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes (OpenAPI contracts and docs/api/attention.md) - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Scott Tong <scott@scottsmbpm5max.lan> Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
2539004201 |
fix(ephemeral): start a restarted volume-free server over the Runner controller files it wrote itself (#20952)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Cloud servers run volume-free (`PAPERCLIP_EPHEMERAL_SERVER=true`). Postgres and object storage hold the durable state. A startup gate refuses legacy volume data that was not migrated. > - The gate reads one family of data from disk: Runner controller files under `runtime/paperclip-runner/durable-sessions/`. It treats a controller file that the database does not prove as legacy data. > - A volume-free server also writes these files during normal work. A warm follow-up restores the previous run's checkpoint as a full `durable.control-plane-state.v2` file. The file stays full until the new controller registers its database checkpoint writer. > - When the server crashes in that window and restarts on the same filesystem, the gate finds its own file and refuses to start. Every later start refuses the same way. This is a full outage that no Retry can fix. > - This pull request gives the gate a durable marker. A volume-free server claims an empty state directory before any controller can write. Files in a claimed directory belong to a volume-free server and do not block startup. > - The benefit is that a crash restart of a volume-free server starts again. Genuine legacy data keeps its protection, and the refusal now names the run, task and file. ## Linked Issues or Issue Description **What happened?** The first restart-chaos waves in volume-free server mode (harness flag `--ephemeral-server`, #20930) found this outage. Wave `w-20261111T212429-s6402` (Claude through ACPX, Daytona warm sandboxes) did a fresh-home deploy and then a `kill -9` of that server. The replacement server exited with `Volume-free server data is not eligible: native_controllers_require_migration`. A second start at +608 s exited the same way. The server stayed down. **Root cause** `hasLocalNativeControllerObligations` in `server/src/services/native-runtime/native-controller-migration.ts` scans every Runner session in the state directory. The kept instance shows the cause. The session's newest run `4a4d7ca1` was `running` (admitted 3 s before the kill). Its controller root held the full `paperclip.runner.durable.control-plane-state.v2` file of its warm predecessor `9c13b7f8`. A controller writes that full file when it opens and replaces it with the `external-control-plane-state.v1` marker only in `setCheckpointPersistence`. `preservedActiveCheckpoint` accepts only the marker, so the gate returned `true`. A Codex wave with the same fault sequence passed because no warm follow-up was in this window at the kill. Claude W2 cells start a new warm follow-up every 15 to 35 s, so the window is open much more often. **Can staging hit it?** Yes, with a restart that keeps the container. Railway documents that a manual Restart "restarts the process within the deployment's container", so the filesystem stays. Railway does not document whether an automatic crash restart reuses the container. A redeploy makes a new container, so it does not hit this. **Expected behavior** A volume-free server starts again after any restart. State that it wrote itself is disposable, because the database checkpoint is the durable record. Data that a volume-backed server wrote keeps its current protection. ## What Changed - Add `server/src/services/native-runtime/volume-free-controller-state.ts`. `claimVolumeFreeControllerState(true)` writes `.volume-free-owner.json` atomically (temporary file, fsync, rename, directory fsync) when the state directory is absent or empty. It returns `owned` for a directory that has the marker and `unclaimed` for a directory with other files. `claimVolumeFreeControllerState(false)` removes the marker. - `server/src/index.ts` calls the claim right after the volume-free configuration check. This is before any controller or recovery can write. A server started without volume-free mode releases the claim, because it may write files that exist only on its disk. An `unclaimed` result logs a warning. - `hasLocalNativeControllerObligations` ignores the marker entries. It returns no obligation for a claimed directory. It now delegates to `findLocalNativeControllerObligation`, which returns the run, task and file (or `database_checkpoint`). - `assertEphemeralServerData` adds that detail to the `native_controllers_require_migration` refusal. The error says which run and file the database does not prove and why startup refuses. - Document the marker in `doc/DEVELOPING.md` ("Runner controller state on a restarted volume-free server"). ## Verification - Red, before the gate reads the marker: `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/native-controller-migration.integration.test.ts -t volume_free_owner` failed with `AssertionError: expected true to be false` at the first gate check. The fixture is a running warm run whose root holds a full `durable.control-plane-state.v2` file, the same shape as the incident. - Green: the same test passes. It checks: claimed directory gives no obligation; release gives the obligation back; a non-empty unclaimed directory stays blocked; the refusal names the run; the controller file is never changed or removed. - `ephemeral-server-data.test.ts`, `ephemeral-server-contract.test.ts` and all 57 cells of `native-controller-migration.integration.test.ts` pass. Server `tsc --noEmit` passes. - Probe on a copy of the kept instance database and the killed server's state directory: the unmodified gate returned an obligation for run `4a4d7ca1` (source file `.../7dbf2d5c.../control-plane/control-plane-state.json`). - Chaos: the s6402 shape (`--ephemeral-server --adapters claude --workloads W2 --faults K1,K2,K15 --k15-fresh-home --seed 6402`) runs on this branch merged with #20930. The result is added in a comment. ## Risks - A filesystem that a volume-free server claimed and that a pre-marker, volume-backed server then wrote is not detected. This needs a downgrade and a mode change on one disk. A current server in volume mode removes the marker, which closes the normal path. - A volume-free server on a directory written before this change (no marker) keeps today's refusal. A new deploy makes a new container, so staging claims the empty directory on first start after this deploy. - Other scanners of the state directory filter on the 64-hex session pattern, so the marker file does not change them. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edit, test runs). Default context window, extended reasoning on. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
855e241887 |
fix(server): record a long event-loop block with the deltas that name its cause (#20709)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - One server process owns the controller leases of its runs, renews them on timers, and serves every request > - On a hosted instance, the process stopped for about 103 seconds without a restart. Every lease lapsed, and two runs needed a manual Retry (the symptom is fixed in #20694) > - Nobody can say what blocked the process. The server records no event-loop delay, and it has no CPU, GC or memory record for that minute > - This pull request logs one local line for each event-loop block of 2 s or more. The line holds the deltas that tell the possible causes apart, and a likely-cause class > - The benefit is that the next stall names its class (main-thread CPU, GC, paging, CPU throttling, host suspend, or a blocking sync call) and its exact time, so we can match it to the run timeline and fix the cause ## Linked Issues or Issue Description Refs #20694 **What happened?** On a hosted instance, the merged run-event timeline of all active runs had no events from 08:19:52.04Z to 08:21:35.37Z (103.3 s). Events from streaming runs arrived about every 0.5 s before and after the gap. The process did not restart (the run diagnostic shows an observer uptime of about 24 minutes when the reaper ran). The first event after the gap was a span with a 60 s timeout that ended after 103.5 s. Thus the timer callback ran about 43 s late. This agrees with a blocked event loop, or with a frozen process. Nothing in the process recorded which one. **Expected behavior** A long event-loop block is a product failure. It lapses every lease and freezes every user. The server must record each long block with enough data to identify its cause. **Steps to reproduce** 1. Start the server. 2. Block the main thread for more than 2 s, for example with a synchronous read on a slow disk or a busy loop. 3. Before this change: the logs show nothing. After this change: the logs show `event loop blocked for <n> ms (likely <cause>)` with the measured deltas. ## What Changed - `server/src/event-loop-stall-monitor.ts` (new): one unref'd 1.5 s timeout, re-armed after each check, measures how late it runs. (Server startup creates no interval timers, so it is not a `setInterval`.) When a check is 2 s or more late, it logs once with these fields: - stall duration, start and end time (wall clock and monotonic clock) - process CPU (user and system), GC pause time and count (`perf_hooks` GC entries) - major and minor page faults, file-system reads and writes, voluntary and involuntary context switches (`process.resourceUsage()`) - cgroup v2 `throttled_usec` delta (Linux; read asynchronously, best effort) - RSS, heap and external memory - `likelyCause`: `host_suspended`, `gc`, `memory_paging`, `cpu_throttled`, `main_thread_cpu`, or `blocked_off_cpu` (sync file I/O, a sync child process, or another off-CPU wait) - `server/src/index.ts`: starts the monitor when the listener binds. It stops the monitor at the start of shutdown, and when a failed startup closes the listener. It writes the line with the existing `logger.warn`. - This is a local log line only. It is not the Telemetry path and not the Observability (OpenTelemetry) path. No data leaves the process. ## Verification - `pnpm --filter @paperclipai/server exec vitest run src/event-loop-stall-monitor.test.ts src/__tests__/server-startup-feedback-export.test.ts` (131 passed; the monitor file passed 5 runs in a row): - an 800 ms busy loop is recorded as `main_thread_cpu`, with the stall and CPU time - an 800 ms `Atomics.wait` (blocks without CPU, like a sync read on a stalled disk) is recorded as `blocked_off_cpu` - nothing is logged while the loop is responsive, or after `stop()` (injected clock) - the file-read delta of a stall is kept while the cgroup read is pending - server startup still creates no interval timer - a 100 s stall with 90 s of cgroup throttling is recorded as `cpu_throttled`, with the stall start and end times - each cause class is selected from its deltas - `pnpm --filter @paperclipai/server exec tsc --noEmit` passes. - CI supplies the broad server suites. ## Risks - Low risk. The timeout reads only process counters. Every 10 s it reads one small cgroup file asynchronously. It does not query the database or the network, so it does not keep an idle instance or its database awake. It is unref'd and stopped on shutdown. - `process.cpuUsage()` counts all threads of the process, not only the main thread. A busy libuv thread pool during a stall can make an off-CPU block look like `main_thread_cpu`. The raw deltas are in the line, so a reader can check this. - If a platform suspends and resumes the process, each resume can log one `host_suspended` line. This is expected and useful. - The monitor names the class of a block, not the call stack. If the class is not enough, the next step is a bounded CPU profile around the stall time. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude Code with tool use, shell, and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes (the module comment carries the contract) - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
|
a46b9ac724 |
fix(native): wait out an ACP usage limit's reported reset, then Retry the task (#20692)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs (Paperclip Runner with an ACP harness, such as Claude)
can stop because the provider account runs out of its usage allowance
> - The ACP harness reports this as a "terminal limit failure".
Paperclip blocked the task behind a board hold that called the session
state ambiguous, and the only way forward was a manual Retry
> - The provider also says when the limit resets, for example "resets
Nov 12, 6pm (UTC)" for a weekly limit. Paperclip read only clock-only
resets ("resets 2am"), so the agent-level wait fell back to one hour.
Each later wake for the agent then hit the limit again and blocked
another task
> - This pull request reads calendar resets, says on the hold when the
limit resets, and requests the task's Retry itself at that time when the
reset is reported and bounded
> - The benefit is that a usage limit no longer needs a person to click
Retry, and the agent's other tasks wait instead of failing one by one
## Linked Issues or Issue Description
**What happened?**
A native ACP run failed with `native_provider_usage_limit` ("ACP agent
reported a terminal limit failure. You've hit your weekly limit · resets
Nov 12, 6pm (UTC)"). The task went to `blocked` with a board-owned hold.
The hold said "Inspect the original provider failure and explicitly
resolve the ambiguous session state". Nothing resumed the task after the
reset. The agent-level `provider_limit_wait` recorded "no reset time was
reported, so Paperclip waits one hour", so every hour another wake
dispatched, hit the limit, and blocked another task. In one 48-hour
window on a staging instance, this blocked 7 tasks for one agent.
**What did you expect?**
When the provider reports a bounded reset, the task waits and continues
after the reset, like the existing `provider_limit_wait` path. When the
reset is unknown, the task fails with a clear message and a working
Retry.
**Steps to reproduce**
1. Run a native ACP agent until its provider reports a weekly or session
limit with a reset time.
2. Look at the task: it is blocked with an "ambiguous session state"
hold.
3. Wait past the reported reset: the task stays blocked until a person
clicks Retry.
Refs #20666 (other recovery work in the same staging census; not changed
here)
## What Changed
- `provider-quota-reset.ts`: parse calendar resets such as "resets Nov
12, 6pm (UTC)", "resets Jul 17th at 4:30 PM (America/Chicago)" and
"Resets on August 2, 9am (UTC)". A past date rolls into next year. An
impossible date (Nov 31) is refused. Clock-only parsing is unchanged.
This also fixes the agent-level `provider_limit_wait` reset and the
legacy `provider_quota` reset for these messages.
- New `native-runtime/provider-limit-resume.ts`:
- `readNativeUsageLimitReset` reads the reset only from the run's own
durable events. The last turn event must be a provider-observed
`turn.failed` with a `rate_limit` or `quota_exhausted` provider failure,
and the reset must be at most eight days out.
- `requestProviderLimitResumeRetry` asks for the task's Retry
(`requestTaskRetry`, `initiatedBy: "provider_limit_reset"`) on behalf of
the user who requested the run, or the run's accountable user for
automatic wakes. It is idempotent per run and records its outcome on the
hold. At most three automatic Retries follow one another without a
successful run.
- `registerProviderLimitResumeWork` is a scoped deadline owner (shared
work scheduler, no polling). Each active usage-limit hold is due at its
reset. Startup rebuilds it from the holds, and the executor notifies it
in the hold's transaction.
- `native-session-executor.ts`: for a terminal usage limit, the hold and
run failure detail say when the limit resets and whether Paperclip
retries then, instead of "ambiguous session state". The evidence records
`providerLimitResetAt`.
- `task-retry.ts`: accept `initiatedBy: "provider_limit_reset"`. Its
activity is recorded as system actor `provider-limit-reset` on behalf of
the user, like `restart_recovery`.
- `doc/execution-semantics.md`: document the wait and its limits.
Previous-release story: holds saved by the previous release carry
forward. At startup the owner reads each active
`native_provider_usage_limit` hold. It rewrites the hold's message with
the reset when the reset is known and still ahead, and it requests the
Retry at the reset. Holds whose reset is unknown, more than a day in the
past, or that have no accountable user keep their Retry and are recorded
as not requested, so they are not read again.
## Verification
- `npx vitest run server/src/services/provider-quota-reset.test.ts` — 4
tests. Three fail on `origin/master` code (calendar resets return
`null`), all pass with this change.
- `npx vitest run
server/src/services/native-runtime/native-provider-capacity.integration.test.ts`
— 5 tests on embedded Postgres:
- the existing usage-terminal test (both replay modes) still blocks with
a Retry, and no automatic Retry is requested without a reported reset;
- "waits for a reported usage-limit reset and then requests the task's
Retry once": the hold says when the limit resets, an old-release hold is
rewritten, nothing starts before the reset, one Retry receipt is created
for the accountable user at the reset, and a second pass does nothing.
Fails on `origin/master` executor code;
- "starts the Retry from the hold's own deadline": a real delivery
coordinator rebuilds the deadline from the hold and calls Retry dispatch
once, after the reset;
- "leaves a visible Retry when no user is accountable for the run".
Fails on `origin/master` executor code.
- Focused regression suites pass (642 tests): `task-retry`,
`quarantined-task-retry`, `heartbeat-native-restart-suspension`,
`native-startup-retry.integration`, `provider-failure-classification`,
`heartbeat-provider-limit-wait`, `deferred-execution-receipt-work`,
`chat-run-publications`, `native-runner-restart-recovery.integration`,
`native-remote-retry-admission`.
- `pnpm --filter @paperclipai/server exec tsc --noEmit` passes.
- `pnpm verify` selected a broad run (exit 2, because the docs file is a
global CI input). CI supplies the broad coverage.
## Risks
- The automatic Retry uses the accountable user's authority for
automatic (system) wakes. It is the same Retry the hold already offered
that user, and dispatch still checks Retry authority, stop proof, saved
work and action reconciliation. It is never broader than a click.
- A provider that keeps reporting resets that do not restore capacity
gets at most three automatic Retries in a row; then the Retry stays
visible.
- The deadline owner reads all active usage-limit holds at startup
(limit 500 rows). The number of such holds is small.
- Not changed: the agent-level wait still holds new wakes for at most
one day, as before.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, read-only staging API reads). Extended
reasoning enabled.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude <noreply@anthropic.com>
|
||
|
|
d0c30e69a9 |
fix(storage): abort a snapshot upload a restart cut off once its save attempt is superseded (#20542)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Remote workspaces save their files as encrypted snapshots in object
storage. Snapshots over 16 MiB use S3 multipart uploads.
> - A server restart can cut off a snapshot upload in the middle of an
`UploadPart` request.
> - The server aborts a multipart upload only in-process, on an error or
an abort signal. A SIGKILL can never do that. On SIGTERM, the drain step
timed out while the store held the part, and the process exited without
`AbortMultipartUpload`.
> - The incomplete upload then stays in the bucket. Nothing records its
upload id and nothing finds it again, so incomplete uploads pile up and
the operator pays for their parts.
> - This pull request aborts the incomplete uploads of each save attempt
when that attempt can no longer win, with no polling and no schema
change.
> - The benefit is that no incomplete upload outlives its save attempt.
## Linked Issues or Issue Description
**What happened?**
A restart-chaos wave (Daytona, R1 workload, storage fault ST4,
`--storage-trigger snapshot-part`, faults K1 and K2, master
`fea4530a30`) passed all 3 tasks. All 3 committed snapshots read back
intact. But the storage audit found 3 incomplete multipart uploads. Each
one was the exact upload that a restart cut off in the middle of
`UploadPart`. The retried save then published a different attempt of the
same snapshot.
**Expected behavior**
When a save attempt can no longer become the snapshot's object, its
incomplete multipart uploads are aborted. The storage audit shows 0
incomplete uploads after the wave.
**Steps to reproduce**
1. Run a snapshot save larger than 16 MiB against S3.
2. Kill the server (SIGKILL), or send SIGTERM while the store holds an
`UploadPart` request.
3. Let the save retry and publish.
4. List the bucket's multipart uploads. The cut-off attempt's upload is
still there.
**Paperclip version or commit**
master `fea4530a30` and later.
**Relevant logs or output**
`shutdown step timed out; continuing shutdown without it
{"step":"delivery_work_stop","durationMs":20001}`, then `Paperclip did
not stop within 30000ms; sending SIGKILL`.
## What Changed
- The S3 storage provider and the storage service get
`abortIncompleteUploads(objectKey)`. It lists the multipart uploads
under that exact key (`ListMultipartUploads` with the key as the
prefix), skips other keys that only share the prefix, and aborts each
one. `NoSuchUpload` counts as already done. The local-disk provider has
no multipart uploads and does nothing.
- Save publication: the save journals every attempt's object key in
`metadata.uploads` before any remote I/O. When one attempt moves the
snapshot from `pending` to `uploaded`, no other attempt can win any
more. The winning process now aborts the incomplete uploads of every
other journaled attempt key. Each abort is best effort and bounded to 5
s. A snapshot with one attempt sends no extra request.
- A losing uploader that is still live: if its upload fails after
another attempt has published the snapshot, it now returns the published
snapshot (after it verifies it) instead of the error. So an abort from
the winner never fails a save that is still running in another process.
- Snapshot retention: before it deletes a retired snapshot's objects
(this includes saves that #20502 retires as abandoned), retention aborts
the incomplete uploads of each attempt key. A store that refuses the
listing (for example a missing `s3:ListBucketMultipartUploads`
permission) logs a warning and does not block the deletion.
- Abandoned saves: when the ownership reconciler retires an abandoned
save (`remote_snapshot_save` and `remote_snapshot_ledger` settlement,
from #20502), it now aborts the incomplete uploads of every journaled
attempt key after its transaction commits. The owner run is terminal and
the save work is settled, so no live uploader can own those uploads.
This is the durable path for SIGKILL: the reconciler pass after boot
settles the dead save and cleans its uploads.
- SIGTERM drain: the multipart upload code keeps an in-process set of
running uploads. Shutdown aborts them right away when
`delivery_work_stop` times out. It also aborts them after the heartbeat
run drain, before the finalizer and workspace-recovery waits, after a 5
s grace for them to finish. It waits at most 10 s for the
`AbortMultipartUpload` calls. Shutdown does nothing extra when no upload
runs.
- No polling and no periodic sweep. All triggers are events that already
exist: publication, abandonment settlement, retention of a retired
snapshot, and shutdown.
Previous-release story (`rules/upgrade-safe-run-state.mdc`): this adds
no persisted state and changes no saved shape. A snapshot that the
previous release saved with cut-off attempts in `metadata.uploads`
carries forward. Its incomplete uploads are aborted when the new release
publishes a retry of that snapshot, or when retention retires the
snapshot.
## Verification
- Red before, green after (all fail on master, pass with this change):
- `server/src/services/remote-workspace-snapshot-storage.test.ts`:
"aborts the incomplete upload of an attempt a restart cut off once a
retried save publishes" and "adopts the published save when another
uploader aborts this attempt's upload".
- `server/src/services/remote-workspace-snapshot-retention.test.ts`:
"aborts each attempt's incomplete multipart upload before deleting a
retired snapshot's objects". A second test proves that a refused listing
does not block retention.
- `server/src/__tests__/storage-s3-multipart-shutdown.test.ts`: file and
streamed uploads whose part the store holds are aborted at shutdown, and
the provider aborts only the exact key across listing pages.
- `server/src/__tests__/remote-snapshot-ledger-settlement.test.ts`:
"aborts the incomplete multipart uploads of every attempt when the
reconciler retires an abandoned save" (both settlement kinds; a key
outside the snapshot's prefix is skipped).
- `tests/restart-chaos/storage-orphan-uploads.test.ts` (MinIO): the real
provider aborts both cut-off uploads of one attempt key, and keeps a
sibling attempt and a longer key that shares the prefix.
- `cd server && npx vitest run src/__tests__/storage-s3 src/storage
src/__tests__/remote-snapshot-ledger-settlement.test.ts
src/services/remote-workspace-snapshot
src/services/remote-workspace-transport.test.ts
src/__tests__/remote-workspace-file-resources.test.ts`: 19 files, 517
tests pass; with `src/services/recovery`, `src/__tests__/ownership*` and
`heartbeat-workspace-busy.test.ts`, 32 files and 580 tests pass.
- `cd server && npx tsc --noEmit` and `node
scripts/typecheck-test-projects.mjs tests/restart-chaos` pass.
- Live proof, Daytona R1, ST4, `--storage-trigger snapshot-part`, K1/K2,
seed 1033, the evidence wave's flags, image
`paperclip-daytona-runner@sha256:05a4a092…` (controllerMatch: match):
- Before (master `fea4530a30`): `w-20261110T130541-s1033`: 3/3 pass, **3
incomplete multipart uploads**.
- `w-20261110T134623-s1033` on an earlier revision of this PR (publish +
retention hooks only): 2 incomplete uploads remained. Those saves were
abandoned, not retried, so neither hook ran. This led to the settlement
hook.
- After (head `8bc8fc66d0`): `w-20261110T141144-s1033`: 3/3 tasks pass,
exit 0, every fired restart proved it cut the held `UploadPart`, the
reconciler settled 3 abandoned saves, shutdown logged one in-flight
abort, and the storage audit shows **0 incomplete multipart uploads**
and 3 committed snapshots that read back intact.
## Risks
- A live uploader in another process can lose its incomplete upload when
a different attempt publishes first. That uploader now adopts the
published snapshot, so its save still succeeds.
- Narrow gap: an uploader that recorded its attempt but had not yet sent
`CreateMultipartUpload` when the winner published, and that then crashes
during its upload, leaves an incomplete upload until retention retires
the snapshot. An S3 lifecycle rule for incomplete uploads stays a good
backstop.
- Stores without `ListMultipartUploads` permission log a warning and
keep the previous behavior.
- Shutdown can take up to 10 s longer, and only when an upload is still
running after the drain.
## Model Used
Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
68253e3175 |
fix(recovery): continue a run refused by an inert cleanup quarantine with no click (#20260)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - When a native session's cleanup fails unrecoverably, the runtime
quarantines it in process. The next run for that task is refused with
`native_session_cleanup_quarantined`.
> - A board Retry or a new message already recovers from this.
`verifyStoppedNativeSessionForContinuation` proves that the old runner
and providers stopped, releases the quarantine and starts a fresh
session.
> - Without a click nothing recovers. The automatic safe-replacement
sweep proves stopped sessions only with the Codex replacement verifier,
which needs the failed run's own active turn. A run refused before
launch never has one, so the sweep skips it on every pass.
> - Restarts must recover on their own. This pull request lets the sweep
use the same continuation proof for a cleanup-quarantined run.
> - The benefit is that an inert quarantine turns into a fresh-session
continuation with no click, and an unproven one still keeps its board
Retry.
## Linked Issues or Issue Description
**What happened?**
In restart-chaos wave seed 9802 (local, G1, Claude), task t02's run
71f03e16 succeeded, but its runner cleanup could not prove it drained
and was quarantined. Follow-up run 0b4b404f was refused before launch
with `native_session_cleanup_quarantined`, and the task showed
"Automatic recovery stopped" until someone pressed Retry.
**Expected behavior**
When the quarantined session's owner run and every provider process are
proven stopped, recovery continues the task by itself in a fresh
session, without charging an attempt. When that cannot be proven, the
board Retry stays available.
**Steps to reproduce**
1. Have a native run succeed while its runner cleanup is quarantined.
2. Let the next run for the task be refused with
`native_session_cleanup_quarantined` before it launches.
3. Stop the old runner and provider processes, then run the
safe-replacement sweep. No successor is created.
**Paperclip version or commit**
master at
|
||
|
|
5a51a7685d |
fix(budgets): raise_budget_and_resume continues the work its budget stopped, once (#20144)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A budget hard stop cancels the scope's running work; the board resolves it with Raise budget and resume > - In the restart-chaos H1 scenario, the cancelled tasks did not continue after resume and each needed a manual Retry > - Resume is the board's explicit decision to continue, so the work it stopped should continue once, as if the board clicked Retry > - This pull request files that Retry automatically, keyed by the cancelled run, once the run's stop has settled > - The benefit is that a budget raise restores the work without manual Retries, without repeating completed work, and without charging the run's attempt budget ## Linked Issues or Issue Description Refs #20143 (bug B; this PR builds on it), #20113 (bug A), #20101 (superseded: split into #20143 and this PR) **What happened?** Restart-chaos waves w-20261108T195944-s42 and w-20261108T210118-s9802 (H1). After `raise_budget_and_resume`, CHA-3's budget-cancelled run (enforcement stop) and CHA-1's run (agent lifecycle pause from the budget hold) stayed cancelled until someone clicked Retry. **Expected behavior** Resuming continues each task its budget stopped, exactly once, with no attempt charged, an activity entry, and no repeat of work that already continued. A run a person stopped stays stopped. **Steps to reproduce** Set a budget below spend while a task runs; let the stop cancel the run; resolve with Raise budget and resume. Before this change, no new run is queued. ## What Changed - New table `budget_resume_claims` (migration 0451). One row per cancelled run, outside the run's result JSON, so a Stop that settles during the resume cannot erase it. - `markBudgetCancelledWorkForResume` (called by `resolveIncident` before the incident closes): claims every task run its budget stopped since the incident opened. That covers runs budget enforcement fenced, including stops still in flight, and runs the agent lifecycle pause stopped for this budget (`pauseStop`, carried from the lifecycle operation's `pauseCause` so a resume before the Stop keeps the link). Runs a person stopped are excluded. Two indexed passes led by `(company_id, status)`. - `deliverBudgetResumeContinuations`: once a claimed run's stop settles, files the resolving board member's Retry (`requestTaskRetry`, idempotency key `budget-resume:<run>`, with the member's session source and the chat Retry callback). The Retry path checks, under the task lock, that the run is still the task's latest and the task is open, so continued or completed work is never repeated. Claim settlement and its activity entry commit together; retryable refusals and errors retry. - Delivery runs in the always-on budget-enforcement worker after each pass's Stop delivery. Its schedule includes deliverable claims and the earliest stale-claim expiry. The execution-continuation worker is a backstop. - `agent.paused` / `agent.resumed` logging is in #20113. `budget.cancelled_work_resume_requested` (incident) and `budget.cancelled_work_continued` / `_not_continued` (task) are recorded here. ## Verification - New `budget-resume-continuation.test.ts` (Postgres, real `resolveIncident` → claims → `requestTaskRetry` → real `resumeTaskRetry` into a queued run): - each task continues once, including a blocked task; repeat delivery or resolve saves nothing new; - a stop still in flight at resume, whose terminal write replaces the result JSON, is still continued; - a 30-task scope is fully continued; - a lifecycle-pause stop caused by the budget is continued, and one caused by a manual pause is not; - continued work, done tasks, a run that finished before its stop, and person-stopped runs (including ones enforcement also fenced) are not continued. - `lifecycle.test.ts`: the pause records its cause on the operation and on stopped runs, including when the budget is resumed before the pause's Stop. `budget-enforcement-worker.test.ts`: delivery after Stop delivery, and a wake at claim expiry. - 287 tests across the affected suites passed. Server typecheck (`typecheck-layers`, low-memory): no errors. - Not run: the H1 chaos rerun (Phase 3 runs it after merge). ## Risks - Resume now continues budget-stopped tasks automatically. A board that wants a task stopped can Stop it again; person-stopped runs are never continued. - A new table and one index (migration 0451). > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test execution). Extended reasoning enabled. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
c4ba3974a7 |
fix(heartbeat): close run admission before a graceful shutdown closes its listener (t09) (#20102)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A graceful restart (K2) drains the control plane: it closes HTTP admission, suspends running runs, and the next boot resumes them > - Run admission stayed open while the HTTP listener closed. Heartbeat's shutdown flag was set only later, inside hot-restart preparation > - A run that finished in that window released its issue and promoted the queued comment wake. That started a new run that had no callback listener > - A remote warm continuation then failed with `local_runtime_api_listener_unavailable`, and the task needed a Retry (restart chaos t09) > - This pull request closes run admission before the listener closes, so the promoted wake stays queued for the next boot > - The benefit is that a comment sent during a deploy continues on its own after the restart ## Linked Issues or Issue Description Refs #19596 **What happened?** In restart-chaos wave w-20261108T192459-s9501 (Daytona warm, Codex C1, K2), a K2 SIGTERM started a graceful drain at 19:42:52 UTC, and the HTTP listener closed first. At 19:43:03 the running turn finished and released its issue. That promoted the queued comment wake, and `startNextQueuedRunForAgent` started a new run about 11 s into the drain. Backend setup resolved the local runtime API URL, found the listener closed, and threw `local_runtime_api_listener_unavailable`. The remote target is excluded from the closed-listener handoff, so the run failed, and the task waited for a manual Retry. **Expected behavior** No queued run starts once a graceful shutdown has begun. A wake promoted during the drain stays queued, and the next boot starts it with no Retry. **Steps to reproduce** 1. Run a remote warm task and post a comment while its turn runs, so the comment wake queues behind it. 2. Send SIGTERM to the control plane just before the turn finishes. 3. Before this change, a new run starts during the drain and fails with `local_runtime_api_listener_unavailable`. **Paperclip version or commit** master at |
||
|
|
e8a3f1f6f2 |
feat(recovery): prove a replaced container's records gone through a fenced controller epoch (#20054)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - After a restart, the control plane settles in-progress records (leases, claims, intents) only once it proves their owner process is gone > - The proof compares host and PID space, so a stamp from a replaced container or a rebooted machine stays unprovable forever, and its records never settle > - Self-hosted users on Docker, Kubernetes and Fly hit this on every redeploy; a Railway deployment with an exclusive volume is covered by `deployment_replaced` (#20026) > - This pull request adds a fenced controller epoch: each boot holds a database lease, and an expired boot is fenced so its records can be proven gone > - The benefit is that work stranded by a replaced container is recovered, without keeping an idle database awake ## Linked Issues or Issue Description Follow-up to #20026 and the Phase 2 durable-ownership data. Related: #19965 (owner stamps), #19883 (process space). No duplicate PR was found. **What happened?** A record stamped by a server process in another container reports `different_process_space`. A dead PID elsewhere is never proof, so the record stays `unprovable` and is never settled. Every redeploy of a self-hosted container leaves such records behind. **Expected behavior** A record owned by a server boot that was replaced (stopped cleanly, or lost its lease and was fenced) is proven gone and settled exactly once. A live boot's records stay with it. An idle server issues no recurring queries. **Steps to reproduce** 1. Run Paperclip in a container and start work, so an environment lease is stamped by that boot. 2. Replace the container. The new container has a new PID namespace. 3. The ownership reconciler reports `different_process_space` (unprovable) for the old lease, forever. **Paperclip version or commit** `master` before this PR. ## What Changed - New `controller_instances` table (migration 0449): boot id, epoch (bigserial), process space, pid, a lease on the database clock, `fenced_at` / `fenced_by_epoch`, `stopped_at`. - `controller-instance-lease.ts`, registered with the shared delivery-work coordinator (`controller-instance-lease` queue, runs during drain): - **Holds work → renews; idle → silent.** One upsert creates the row on first use and extends the lease only if the boot holds work: a stamp within the last 90 s, or a stamp whose ownership write may still be pending (held until the server's tracked work settles, capped at 10 minutes; a renewal that sees some other held record does not clear it, so a slow ownership write cannot be outlived), or an in-progress record held by its boot id. Those records are active environment leases, in-flight cleanup claims, sandbox creation intents, workspace finalize attempts, agent lifecycle steps, native finalizer and cleanup leases (`lease_holder` bound to the token), open native coordinator claims, and running legacy runs. Renewal runs every 30 s while the boot holds work. When it finds none, the lease lapses and the worker sleeps until the next stamp (`onControllerStamp`). - **No idle trailing statements.** When the server's last tracked work settles, the worker re-checks holdings at once, at most every 5 s (the rate limit also absorbs the worker's own run). So the last check happens with the work, not 30 s later. - **Quiesce, don't crash.** Each renewal has a 10 s bound. A failed or timed-out renewal applies an internal task drain, so no new work is dispatched. A recovered renewal resumes. The boot exits only after 60 s of failure (counted from the first failed attempt's start) while it still holds work, or at once when it finds itself fenced. - **Fence on demand, by a newer epoch.** For a stamp from another host or PID space, the prover reads that boot's row: - `stopped` or `fenced`: gone. - live lease: alive. - otherwise, the boot's last sign of life is the later of its lease expiry and the record's last write plus the TTL. Within 30 s of that, the owner stays `alive` (`controller_lease_grace`), so the settle watch keeps rechecking. After that, the prover fences the boot, but only with a lease row of its own (created on demand) and only an older epoch. - A graceful shutdown records `stopped_at`. - A reconciliation pass that settles nothing (`record: false` or `settle: false`) never fences and never creates the boot's row. - `owner-proof.ts`: these proofs apply only to server-held records. A stamp with a recorded process group stays unprovable, because agent processes can outlive their server. The reconciler passes each record's `updatedAt` to the proof. - `task-admission.ts`: - the quiesce drain never replaces an operator's drain, and has no idle owner, so HTTP stays open; - a `subscribeIdleWorkSettled` hook fires when the last tracked work finishes. - Off on a Railway deployment with an exclusive volume (it has its own replacement proof). `PAPERCLIP_CONTROLLER_INSTANCE_LEASE=on|off` overrides this. A stateless Railway service keeps the lease. A server without its own lease row never fences. - `maintenance-server-routes.test.ts`: in the post-run quiet window, the lease may check once more that the run's records were released. The test now excludes those statements, then asserts that the lease issues nothing in a further window. - Docs: `doc/execution-semantics.md`; the Phase 2 data doc records the two known gaps. ## Verification - `controller-instance-lease.test.ts` (15 cases): - a replaced container with all Railway variables empty is fenced and settled exactly once over two passes; - a live lease is alive; - an owner in grace, and work written after an idle lapse, stay alive and watched; - an older epoch never fences a newer one; - no fencing without its own lease row (off or unregistered), and an idle prover creates its row on demand; - a stopped boot is gone; - process-group stamps stay unprovable; - idle issues zero statements, and one upsert renews only while records are held; - **a 45 s outage:** quiesced, no exit, resumes; - **a renewal that never answers:** bounded, and the boot exits on time; - **a 70 s outage:** exits at 60–65 s, and the successor recovers the work exactly once; - fenced: stops; - the settle re-check; - graceful stop; off on a volume-exclusive Railway deployment, on for a stateless one, and the on/off override; - a running legacy run keeps the lease live. - Red, with the proof's instance check removed: the replaced-container, live, stopped and successor cases fail. - Wider: task-drain, idle-sleep, idle-local-work, owner-proof and ownership suites, plus the maintenance quiet-window tests. Server low-memory typecheck is clean. ## Risks - **Clock skew.** The proof compares a record's `updatedAt` (often the writer's clock) with the lease (database clock). The 30 s grace absorbs ordinary skew. - **A new exit path exists.** A server whose database is unreachable for more than 60 s while it holds work now exits, after first quiescing. This is the intended safety backstop. - **Bounded trailing activity.** A lease that holds work issues one statement every 30 s, plus at most one check when its work settles. An idle server issues none. - **Boots from before this release** have no row, and their records stay unprovable. The board "mark owner gone" action for those follows in a separate PR. ## Model Used - Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
988a4b6749 |
feat(recovery): ownable-kind registry and report-only ownership reconciler (#19861)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A control-plane restart leaves in-progress records owned by dead processes, and about 31 per-subsystem proofs decide when to release them > - The durable ownership plan (section C) asks for one registry of ownable record kinds and one reconciler > - Phase 1 must measure first: how many in-progress records exist, who owns them, and which invariants they break, with no behavior change > - This pull request registers 46 kinds from the Phase 0 inventory's section C and adds a report-only pass to the existing native finalization reconciler > - The benefit is per-kind invariant-violation counts from chaos waves and staging. That data unlocks Phase 2 (moving kinds onto one settle path) and the removal of the company-wide admission gate ## Linked Issues or Issue Description Refs #19831 (plan, section C and Phase 1). Refs #19846 (Phase 0 inventory, section C). Stacked on #19853 (the prover) and #19851 (one process identity). **Problem** No single place lists the control plane's in-progress records or checks the plan's invariants. Each subsystem releases its own records by its own rules, so nobody can count how many records break "owned", "settled" or "visible next action". **Expected behavior** One report-only pass lists every registered kind, classifies each owner as gone, alive or unprovable, and records each invariant violation once. It changes no record and stays quiet when nothing is in progress. **Impact** Phase 2 moves kinds onto the shared settle path in the order these counts show. Phase 3 removes the admission gate once the I3 data exists. ## What Changed - New `server/src/services/recovery/ownership-registry.ts`: 46 `OwnableKind` entries (`id`, `obligation`, `storage`, `listInProgress`, `settle: null`, `invariantChecks`). Each record names its owner as a process stamp, a fence token, a run, an operator, or none. The skill-cache lock is registered with a named coverage gap because its lock files live in per-skill roots. - New `server/src/services/recovery/ownership-invariant-report.ts`: one pass classifies every owner with `proveOwnerGone` (or the owning run's status), then runs: - I1 owned: an in-progress record with no owner; - I2 settled: an owner proven gone on two consecutive passes while the record is still in progress; - I3 admission independence: terminal runs whose accounting outlives the 10 min closure, which hold company admission; - I4 exactly-once intents: wake requests ended `skipped`/`failed` on a temporary refusal (heuristic until section E); - I7 visible next action: an agent-assigned `in_progress` issue with no live run, no pending wake and no open recovery action for 5 min. - I5, I6 and I8 are reported as named coverage gaps; I8 comes in the speed-budget PR. - Violations are written once per process to the activity log as `recovery.invariant_violation` (`actorId: ownership-reconciler`, `details.mode: report_only`), insert-only with no live-event or plugin fan-out. Nothing goes to Telemetry (AGENTS.md §5.7). - `native-finalization-reconciler.ts`: `registerOwnershipReconciliationWork` runs the pass in the shared delivery-work scheduler (new queue `ownership-invariant-report`). There is no second loop. - `index.ts` registers it after residual recovery and does not await its first pass, so readiness is not delayed. ### Sleep-aware trigger (rules/sleep-aware-background-work.mdc) - **Trigger:** one pass at registration (startup reconstruction from durable truth). After that, a settled `active-execution-recovery` signal, which run lifecycle, controller renewal and process-evidence writes already emit. Signals are coalesced to at most one pass per 30 s. No producer gained a write. - **Pending predicate and next deadline:** a 30 s follow-up only while the last pass saw a changing in-progress set, or an owner proven gone that needs the second I2 pass. - **Quiet:** an empty set or the same stuck set twice sets no deadline. The test asserts `nextWakeAt() === null` after an idle pass. - **Turn path:** the pass never runs before `turn.start`. Stamps read only cached identity. ## Verification - `npx vitest run src/__tests__/ownership-reconciliation-report.test.ts` (embedded Postgres, in `server/`): 4 tests pass: - every kind is registered once with no settle path; - verdicts per class: alive (this process's claim), gone (lease of a failed run), unprovable (no owner; another PID space); - each rule fires on its fixture, and I2 fires only on the second pass; - every touched table is byte-identical before and after (report-only); - activity rows are written once per violation, and none are written with `record: false`; - the worker is quiet when idle and schedules a follow-up only while records change. - `npx vitest run src/__tests__/delivery-work-coordinator.test.ts src/services/owner-proof.test.ts src/__tests__/server-startup-feedback-export.test.ts`: pass. - `npx tsc --noEmit -p server`: no errors. ## Risks - No record changes, so no settle risk. The risk is load: one pass runs about 46 small indexed or `limit 201` queries, at most once per 30 s while runs are active, and none when idle. - I4 matches refusal text with a heuristic. It can over- or under-count until section E gives wake requests closed terminal reasons. It is labelled as such in the code. - Violations dedupe per process, so a restart writes a still-open violation again (once per boot). Chaos reports count distinct `(rule, entityId)`. - Previous-release story (`rules/upgrade-safe-run-state.mdc`): no persisted run state changes. The pass reads rows of any age, and a missing field classifies as unprovable. - `heartbeat_runs` and `native_run_finalizations` record no host or PID space, so foreign-boot owners there classify as unprovable (`missing_identity`). This is the inventory's finding 2, and the report makes it countable. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
795ad16c1f |
fix(server): bound the shutdown steps before the run drain (#19165)
Bound initial shutdown waits and finish workspace repairs before teardown. Share a 20-second budget between waits before the heartbeat run drain. Retain the original workspace recovery stop promise and join it after the run drain and finalizers, before application and database teardown. Keep that join for hot restart when the run drain is skipped. Log step and drain durations and bound telemetry flush separately after the drain. Verification: all 154 tests across the three relevant shutdown/startup suites and every server source/test typecheck layer passed. Both actual server lifecycle variants reproduce premature teardown without the fix. All applicable CI passed and exact-head Greptile Apex scored 5/5. Co-Authored-By: Devin Foley <devinfoley@users.noreply.github.com> Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
|
e2c7d81067 |
fix(server): keep reply recovery off web startup critical path (#19769)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The server restores retained native replies during startup. > - Fair reply recovery can admit more due work while an earlier group remains active. > - Waiting for that whole pass can hold web readiness on unrelated retained replies. > - This pull request tracks reply recovery with the existing background startup owner. > - The server can become ready while idle and shutdown still account for that work. ## Linked Issues or Issue Description Refs: #19726. **What happened?** The startup path awaited the native response worker's first pass. That pass can remain active while fair dispatch admits later replies. A controlled `startServer` regression stays in recovery until the held reply pass finishes. A deployed backlog supported investigating this boundary, but no captured runtime stack proves it caused the observed startup delay. **Expected behavior** Retained reply recovery must keep its authority checks, retry deadlines, and fair dispatch. Its completion must not be a condition for unrelated web readiness. Active recovery must still block idle and be joined at shutdown. **Steps to reproduce** Run the startup regression with a held native response worker promise. Before this change, `startServer` does not complete. After this change, startup reports ready, idle work remains active, and shutdown waits until the promise settles. ## What Changed - Track the initial native response pass with the existing heartbeat scheduler work owner. - Extend the actual startup wiring test to cover readiness, idle accounting, worker stop, and shutdown join. - Document the response recovery boundary for idle and readiness. ## Verification - Mandatory agent preflight passed on macOS arm64 with Node 26.4.0. - True RED: the new startup case failed because `startServer` had not completed while reply work was held. The same case passed after the change. - Final head `279e97587b` passed the full startup test file: 123/123 tests. - The unchanged real PostgreSQL later-page case passed. It holds 100 old publication rows, publishes the independent 101st reply once while the first pass remains pending, then verifies completion and no recurring idle SQL. - Final-head CI passed: 60 successful checks and 3 skipped checks, including `ci / verify`. Fresh Greptile is 5/5 with no open threads. - Server TypeScript check passed: `node node_modules/typescript/bin/tsc -p server/tsconfig.json --noEmit`. - Initial test-launch and missing generated SDK setup failures were resolved before the regression ran. They are not RED or pass evidence. - No deployment or live recovery action was performed for this PR. Focused commands, from `server/`: ```sh node ../node_modules/vitest/vitest.mjs run src/__tests__/server-startup-feedback-export.test.ts --maxWorkers=1 --no-file-parallelism node ../cli/node_modules/tsx/dist/cli.mjs ../packages/db/src/test-postgres-command.ts -- node ../node_modules/vitest/vitest.mjs run src/services/native-runtime/native-response-publication-retry.test.ts -t 'publishes a later-page reply while older publication groups remain contended' --maxWorkers=1 --no-file-parallelism --hookTimeout 90000 ``` ## Risks Web readiness can complete before retained reply recovery finishes. The existing scheduler still owns errors, retries, scope checks, and delivery authority. The existing idle tracker and shutdown join retain in-flight work. Other startup recovery gates stay in place. ## Model Used OpenAI Codex, `gpt-6.1-sol`, with reasoning, tool use, and code execution. The context window size is not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
a894b11a2b |
fix(runner): retry a rejected runner suspension and log every cleanup quarantine (#19653)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run's close must prove that its runner durably suspended
before the session's next run may start. If it cannot, the runtime
quarantines the session (`NativeSessionCloseUnrecoverableError`)
> - In a restart chaos storm, an ACPX Claude run succeeded and its
normal close quarantined its own session. The follow-up run on that
session failed 2 s later with `native_session_cleanup_quarantined`, and
no control-plane restart came between the two runs
> - The close sent `runner.suspend` once, the ACPX sidecar rejected it
right after the turn completed, and the barrier gave up at the first
rejected command. No log line showed the quarantine or its cause
> - This pull request retries a rejected suspension within the close
deadline, and logs every cleanup quarantine with its cause when it is
created
> - The benefit is that a successful run no longer blocks its own
follow-up because of a one-time suspension rejection, and any remaining
quarantine can be traced
## Linked Issues or Issue Description
**What happened?**
Run A (ACPX Claude) succeeded. The run log of its close shows:
- `runner did not prove durable suspension before checkpoint`
- `native_session_settlement_incomplete
{"runnerSuspended":false,"providerDrained":true,"suspensionState":{"commandStatus":"failed","runnerLifecycle":"suspended",...},"semanticTools":{"pending":[]},"finalProviderState":{"activeProviderTurnId":null,"providerSettled":true}}`
The runner journal shows `runner.drain` completed, then `runner.suspend`
failed with `failed to suspend ACPX provider: ACPX sidecar command
session.suspend was rejected (retryable=false,
classification=unclassified)`. The follow-up run B on the same session
was created 200 ms after A finished and failed 2 s later at admission
(`retryQuarantinedSessionCleanups`, operator recovery branch). The
server log had no line for the quarantine. One Retry completed the task.
**Steps to reproduce**
1. Run an ACPX task whose sidecar rejects the first `session.suspend`
after the turn completes.
2. Start a follow-up run on the same session.
**Expected behavior**
The close asks again, proves suspension, and the follow-up runs. If a
quarantine is still needed, it is logged with its cause.
**Actual behavior**
The close fails on the first rejection, quarantines the session, and the
follow-up fails with no logged cause.
## What Changed
- `runner/.../durable/runner.rs` (`process_command`): a failed
`runner.suspend`, on first execution or replay, now keeps the runner in
`Continue` instead of `Suspend`. Before, the runner recorded `suspended`
over a provider whose session was still open, then exited. The RC-34
journal shows exactly that: runner lifecycle `suspended`, ACPX provider
`session_open`. A failed `runner.shutdown` still stops the runner.
- `durable-prp-control-plane.ts`: a failed or rejected `runner.suspend`
now lifts the connection's terminal-lifecycle fence, so the next pending
command (the fresh suspend) is delivered on the same connection.
Completed suspensions and every shutdown keep the fence. New test:
"delivers a fresh suspension after a rejected one on the same
connection". Before the fix, the retry was never sent and the test timed
out waiting for the frame.
- `runnerd-codex-transport.ts` (`awaitRunnerSuspensionBarrier`): after a
rejected `runner.suspend`, the barrier sends a fresh suspend command, up
to 3 attempts, 250 ms apart, inside the same close deadline. An exited
runner or a spent budget still fails closed. A rejection changes no
state: the runner returns before it records suspension.
- `native-session-runtime.ts`: new
`onNativeSessionCleanupQuarantined(listener)`. It is called when a
cleanup quarantine is created or escalates to operator recovery, with
company, run, issue, session, `operatorRecoveryRequired`, a bounded
cause (error name, code, message) and bounded settlement facts.
- `server`: `logNativeCleanupQuarantines` logs each quarantine as a
warning. The subscription is process-wide and installed once, so closes
that fail during shutdown, after HTTP closes, are still logged.
## Verification
- Red before the change: `pnpm --filter @paperclipai/paperclip-runner
exec vitest run src/live/runnerd-codex-transport.test.ts -t "rejected
suspension"`.
- "retries a rejected suspension…" got `false`.
- "stops retrying…" saw 1 attempt instead of 3.
- Green after the change: both pass.
- Rust: new test
`rejected_suspend_keeps_the_runner_serving_its_controller` covers a
failed suspend, its replay, and an accepted retry with a fresh command.
`failed_lifecycle_commands_replay_their_terminal_transition` now expects
`Continue` for suspend and `Shutdown` for shutdown. The `pnpm run
check:runner` unit suites pass (559). `acpx_park_readiness` has two
timing tests (`< 500ms`) that fail at this host load on unchanged master
as well.
- New runtime test: "announces every cleanup quarantine with its cause
when it is created" (operator-required quarantine, cause and settlement
reported once). `src/native-session-runtime.test.ts` passes (212).
- The suspension cases in `runnerd-codex-transport.test.ts` pass, except
"proves local suspension after an event backlog… (1024 suffix deltas)".
That case also times out on unchanged master at this host load (27–35);
I confirmed this by stashing the change and rerunning.
- Server `tsc --noEmit` passes after the runner package build.
## Risks
- A provider that keeps rejecting suspension now gets three tries
instead of one, within the existing close deadline. The final outcome
and quarantine are unchanged.
- The cause of the ACPX sidecar's first rejection is not proven. The
logged evidence fits the sidecar's "not at a safe suspension point"
guard (`turnId`, tools or inputs still set) racing the just-completed
turn. This change makes close tolerate it. It does not change the
sidecar.
- Log volume: one warning per quarantine.
- The Rust change ships in the runner binary. Hosted images need a
rebuild to pick it up.
- After a failed suspend, the runner stays alive with its provider. If
every retry fails, close still treats suspension as unproven and kills
the owned runner process, exactly as before.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
Refs #19532, #19609.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
785ead5c7a |
fix(tasks): hold a Retry while the task's own native session close is retained (#19609)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - When a native provider session close fails, the runner keeps a cleanup owner or quarantine in process memory. Its admission gate refuses new sessions for that task (scoped to the task since #19532) > - The server could not see that state. A task's Retry was accepted (202), its run was dispatched, and the runner refused it at once with `native_session_cleanup_quarantined: prior session cleanup remains incomplete` > - The user saw a Retry that failed instantly, and nothing said what to do > - This pull request lets the server read the runner's per-task cleanup gate and hold the Retry with a clear reason. The Retry starts automatically when the close is released > - The benefit is that a Retry is never dispatched only to fail on a block that is already known ## Linked Issues or Issue Description **What happened?** After two hard restarts, a Codex run's close stayed incomplete. The board Retry for that task was accepted, and its run failed in about 1 s with `native_session_cleanup_quarantined: prior session cleanup remains incomplete`. The next Retry did the same. **Steps to reproduce** 1. Let a native session close fail or stay pending for a task. 2. Press Retry on that task. **Expected behavior** The Retry waits with a clear reason, and starts when the cleanup finishes. If the cleanup needs an operator, Retry is not offered and the one working action is shown. **Actual behavior** The Retry is accepted and its run fails at once. ## What Changed - `native-session-runtime.ts`: - New `nativeSessionCleanupBlock({ companyId, issueId })`. It is a read-only view of the in-process admission gate for one task: `pending` (a retained close or a bounded automatic close recovery still owns the task's provider resources), `operator_required`, or `null`. It grants no authority. - New `onNativeSessionCleanupSettled(listener)`. It is called whenever a cleanup owner or quarantine entry is released. - `explicit-native-continuation.ts` (the continuation gate for a Retry or a new message): - `pending`: the input is held with `local_cleanup` ("Retry is waiting for the previous execution to finish cleanup. Your message is saved."). It starts automatically when the close is released. - `operator_required`: handled exactly like a recorded `native_session_cleanup_quarantined` run. Retry is not offered. A new message runs the existing stopped-process proof (`verifyStoppedNativeSessionForContinuation`), which retires the runner quarantine when the old processes are proven stopped. - `native-cleanup-release-wake.ts` (registered in `server/src/index.ts`): each cleanup release wakes deferred execution for the company in a transaction, so a held Retry re-evaluates. - `server/src/vendor/paperclip-runner/index.ts`: exports the two runtime functions. ## Verification - Red before the gate (server change stashed): `pnpm --filter @paperclipai/server exec vitest run src/services/quarantined-task-retry.test.ts -t "holds a Retry"` dispatched a successor run while the task's close was retained (`expected [ { … } ] to have a length of +0 but got 1`). - Green after the gate: - The Retry is held (`deferred_issue_execution`, `executionWait.reason = local_cleanup`), and no successor is dispatched. - A different task is not blocked. - When the close settles, the release is announced, and the next pass dispatches exactly one successor. - The whole `quarantined-task-retry.test.ts` file passes (28 tests). `src/native-session-runtime.test.ts` passes. Server `tsc --noEmit` passes after the runner package build. - Live chaos replays: results are added in a comment. ## Risks - The gate reads process memory, so it applies only where the server and the runner session share a process (local runner). Other deployments keep today's behavior. - A held Retry re-evaluates on the release wake (sent in its own transaction) or on the existing 30 s deferred-execution pass. When the bounded automatic close budget is spent with nothing else scheduled, the view reports `operator_required`. Retry is then no longer offered, and a new message reaches the stopped-process recovery instead of waiting forever. - No schema or API shape changes. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Refs #19532. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
124b2cd0ad |
fix(server): certify background deadlines for idle tenants (#19001)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Background services own future checks, retries, and recovery work. > - An idle tenant can stop only when a durable wake covers its remaining duties. > - The current alarm catalog covers several domains but leaves these owners blocked. > - A saved future timestamp alone cannot prove that reconstruction or admitted work has settled. > - This pull request binds durable domain snapshots to the actual registered worker revisions. > - The tenant can hand off supported future duties and retain all checks for uncertain or active work. ## Linked Issues or Issue Description No matching public issue was found. Related work: [domain deadline handoff #18656](https://github.com/paperclipai/paperclip/pull/18656), [owned blocker diagnostics #18632](https://github.com/paperclipai/paperclip/pull/18632), and [completed history proof #18646](https://github.com/paperclipai/paperclip/pull/18646). The separate [native idle-session change #18988](https://github.com/paperclipai/paperclip/pull/18988) covers runtime retirement. [Far-future retry re-probes #13534](https://github.com/paperclipai/paperclip/pull/13534) changes recovery responsiveness after an upstream success; this change preserves existing retry decisions. **What happened?** Future status-card, external-object, skill-sync, watchdog, retry, and quota duties lacked complete owner-bound coverage in the idle deadline proof. A completed quota task could also retain its monitor policy and timestamp and remain blocked after the scheduler had disarmed it. **Expected behavior** A settled registered owner can hand off its exact supported future deadline. Active work, dirty reconstruction, transaction uncertainty, changed revisions, claims, unknown state, and unsupported duties keep the tenant awake. Completed history retains its data and stops recurring work when the owner has no remaining duty. **Steps to reproduce** 1. Produce a supported future duty through its normal service. 2. Let the registered worker finish reconstruction and admitted work. 3. Request an owned idle report with the negotiated domain deadline catalog. 4. Check the saved earliest deadline and generation before durable alarm registration. 5. Stop and reconstruct the worker. Check that ordinary admission and claims resume useful work once due. 6. Complete a quota task. Keep its saved policy and timestamp. Check that linked unfinished work still blocks sleep and that settled history does not. **Paperclip version or commit** The owned patch was qualified at `699c8569fa34338dd75cdc5731381ec1e2efdb45`. Publication head `491fac368ee72eaac8ae105d53e6bf6b305d8a6a` normally merges actual master `039e0dad2c51c80251e1302a19c5566e039c357a`. All 43 owned files are byte-identical. All 29 new upstream paths are preserved. The later normal current-master integration is `8c912e1ba4fc66397a7341ade59373af8075068f`, described separately below. **Deployment mode** Source checkout with real PostgreSQL 17 and a controlled local Node process. No deployed alarm, tenant suspension, database suspension, or fleet result is claimed. ## What Changed - Add registered-worker reconciliation revisions and invalidate them on stop. - Keep status-card, external-object, and skill deadline interpretation with their owners. - Reuse the watchdog classifier against the supplied database snapshot. - Validate the exact saved retry run and paired queued wake. - Validate legacy quota source bindings and the current assignee or reviewer policy. - Preserve common issue scope, dependency, sibling, and state checks for covered waits. - Compose normalized owner snapshots within the existing bounded deadline proof and recheck revisions after inspection. - Bind actual service and worker instances from server startup. Missing bindings preserve existing blockers. - Retain completed quota policy and timestamps while independently rejecting linked unfinished runs or wakes. - Block unconsumed budget evaluation inputs through the owning queue predicate. Accounting policy and settlement stay unchanged. - Bind completed quota history through fixed-size UUID-array parameters, preserve exact company/task/source associations, and guard external references before the run-ID join. - Batch watchdog subtrees and shared classifier facts without changing the 64-query or two-second proof budget; preserve exact company pairing for first-run evidence. - Extend production-path regression controls and document supported coverage and limits. ## Verification Current integration head: `688abc6cac6016b9fb50e5832f81e2975ed5d49f`. Fresh exact-head checks completed with **58 successes, three intentional skips and one neutral review**, including aggregate `ci / verify`. The complete current Greptile review is **5/5**. The complete current Ponytail verdict is **Ship (4/5)**, with only optional cleanup of an unused MCP proof input; both prior correctness findings are resolved. All review threads and nested pages were inspected. The normal master merges preserve the registered retry deadline owner and current startup completion log after the same readiness await. The earlier focused evidence below remains tied to its tested source revisions. Current-master integration `8c912e1ba4fc66397a7341ade59373af8075068f`: - Normally merge actual master `46fb933b0155d183ce64a8bdc4950d1b58c926f6`. Every nonoverlapping tree blob retains its owning parent across 14,044 paths. All 42 owner-only files and 15 critical binding/proof/readiness files remain byte-identical. Three automatic overlaps preserve the existing watchdog binding, conservative unavailable-reader fallback and optional actual owner arguments. The two scoped-deadline conflicts retain the entire upstream resource-reservation/mid-pass admission/abort algorithm inside the existing reconciliation revision/running/failure fence, with both parent test sets. - Actual TypeScript 7.0.2 server noEmit passed in 9.434 seconds, normal source declaration layers in 12.102 seconds, and four selected test/helper/config layers extending the actual CI configuration in 3.167 seconds. No compiler option weakening, heap override, donor declarations or SDK replacement stub. - Focused PG0 controls passed: 102 scheduler/proof/owner controls in 16.451 seconds, including all 21 scoped-deadline cases; 121 startup controls; and all 107 unchanged instance-settings route cases in 6.456 seconds. The latter first hit a local prewire-guard listener-registration race. Only the ignored guard was corrected to register the same actual OS-bound server port immediately after native listen returns. Closed, unknown, foreign, offhost and PG destinations remain blocked. Original failure evidence is retained; product code and assertions were unchanged. - The exact current UI policy command now reports zero findings (0.362 seconds); its seven scanner controls pass. The earlier dependent PR used an old scanner/source baseline under a newly enforcing trusted workflow and reported 397 findings. Carrying the genuine current master resolves that mismatch without editing unrelated UI sites or relaxing the gate. - Every meaningful worker checked its executing PID/thread/preload hash and matching receipt. PG0 controls opened no PostgreSQL sockets; all owned processes exited. Original dependency-cache/DNS, ignored-harness syntax/type and guard failures remain separate from product results. - This is fresh semantic and focused runtime qualification, not a local whole-repository or current-head PostgreSQL acceptance rerun. The upstream workspace-busy parked-retry/terminal-holder notification path remains unchanged by this merge and needs its separate current-graph producer check. Historical six-case/scale results below stay tied to their measured sources. A matching controlled default-cadence follow-up is in #19191; no fresh current-head two-cycle or deployed suspension claim follows from this merge. - All 59 exact-head check contexts now pass (55 success, one neutral, three intentional skips), plus the legacy security status. The complete edited Greptile review is 5/5 for this exact commit, with all root/nested pages inspected and no unresolved threads. Ponytail stopped without completing its new turn; it supplied no new verdict or finding. Prior actual findings remain addressed. Earlier scores/checks below belong to their explicit earlier commits. On frozen clean merged source `699c8569fa`: - Real PostgreSQL acceptance: **6/6 passed**,83.483 seconds wall. Actual retry producer creates the exact saved run/wake; stopped and reconstructed owner resumes through current admission, one durable start, real local process output and normal task completion. Includes cancellation, drain arrival, retained quota history with queued-run/wake negatives, watchdog review settlement and budget-input empty/pending/consumed control using the actual owner. - Affected coordinator, deadline, scoped-deadline, recovery and composed-proof controls: **78/78 in five files passed**,32.114 seconds wall, worker 1/no file parallelism. - `pnpm exec tsc --noEmit -p server/tsconfig.json`: **passed**,17.467 seconds. Native TypeScript 7.0.2 and Node 24.22.2. - PostgreSQL tests use one dedicated test database, matching fixture/admin URLs, pool limit four, one worker, and no embedded fallback. Every run recorded unchanged source hashes and a clean checkout. - The eight-second real-clock retry is a domain execution check. It is below the Cloud minimum wake lead and is not a positive accepted hosted sleep cycle. - The completed quota-history product failure and original fixture/protocol failures were preserved; the owning-reader fix passed a real PostgreSQL red/green regression before the fresh merged six-case pass. - After runtime completion, the dedicated database had zero other clients and no owned runtime processes. All three qualification process PIDs exited. The coordinator independently audits and drops the database. - Comment-only cleanup has equal strict TypeScript parser ASTs across 15 files. The final lexical audit finds no added comments relative to actual master. `git diff --check` passed. - Earlier retry fixtures omitted responsible-user membership, advanced JavaScript time past the real PostgreSQL lease clock, or produced stdout without normal task completion. Corrected fixtures use real membership, an eight-second real deadline, and ordinary task comment/completion. The last case exposed legitimate disposition repair rather than duplicate retry promotion. - The quota and watchdog controls first ran before admitted work or the existing 30-second reconstruction duty settled. The stronger tests require an interim blocked report and then exact settled-owner proof. Production deadlines were not changed. The separate completed-quota-history defect was fixed in its reader and verified with linked unfinished-run and wake controls. - The status/object and skill owners reviewed the actual final startup, route and post-backup bindings. Their domain source evidence carries where token-equivalent. The master coordinator coalescing change was reviewed separately. Historical domain PostgreSQL results are not presented as fresh full-suite results on this head. The later master merge adds retry predecessor cleanup checks under the existing issue claim lock. The source audit confirms unchanged saved deadlines and run/wake identities. The three actual retry lifecycle/cancellation/drain cases passed on the batched candidate with this parent (31.817 seconds). Those local dependency controls preceded the current exact-head remote run. Native backup preservation and TestOps changes from master are retained. No pass is inferred for changed dependency behavior solely from unchanged owned files. Review repair published as `643f1e7400`: - The original reader returned `unknown` after seven actual watchdog reviews settled across seven companies. The unchanged composed-report regression passed after batching; the final reader passed in 61.841 seconds, including active queued-work and stopped-owner rejection. - A separate two-company red/green control rejects terminal-run context from another company as first-run grace evidence. Shared facts are serialized once, with per-root IDs and review summaries. - The affected recovery, proof and diagnostic unit controls passed 47/47 (14.575 seconds). Startup owner-contract fixtures passed 121/121 (10.589 seconds). - Diagnostic PostgreSQL controls passed7/7 (12.406 seconds), including the normal budget-input producer and its exact blocker reason. Initial empty-state setup uses the same cleanup as per-case teardown when a dedicated test database is reused. - Actual repository `pnpm exec tsc --noEmit -p server/tsconfig.json` passed 12.710 seconds with no heap override. Final source has no added code comments; whitespace checks pass. - The original query-budget failure, two-company discriminator, initial cross-suite fixture failure and an interrupted incorrect local-driver selector are retained separately. The interrupted run is not a pass. Independent cleanup found zero owned clients or runtime processes. Quota-scale repair published as `6696a37f8a`: - The exact 10,000-row completed-history case reproduced `MAX_PARAMETERS_EXCEEDED`:90,002 bound parameters exceeded the driver's 65,534 limit, and the composed report returned `unknown` despite settled owners. Original failure and actual driver diagnostics are retained. - Fixed pending-work queries use three typed UUID-array parameters plus the result limit, keeping exact company/issue/source tuple bindings. The primary run-ID column stays uncast; a guarded CASE validates the raw external reference before conversion. The 10,000-row cap and 64-query/two-second proof budget are unchanged. - The same 10,000-row composed-report case passed 18.860 seconds, with unfinished run/wake rejection, malformed-reference safety and conservative 10,001-row overflow. Its fixture derives retained history from the actual quota producer and normal task completion, expands valid persisted issue/source pairs, then starts the actual registered owners. - The existing actual quota producer case passed 14.187 seconds, including exact future deadline, retained completion history, unfinished work, real foreign-company source references and noncanonical/malformed references. - Affected owner/proof controls passed 22/22 (26.444 seconds). Production server noEmit passed 16.039 seconds. - CI on 643 identified TS7034/TS7005 in the settled-watchdog fixture's initially untyped collection. It is now explicitly typed. A separate affected-entry semantic check extending the repository's actual `server/src/__tests__/tsconfig.json` passed 15.093 seconds: six affected tests, two owner files, ambient declarations and current imported source, with unchanged strict/module/lib/path settings. The production server config excludes that test directory and is not presented as test-layer proof. No custom suppression or replacement SDK stub was added. - All runs were sequential with pool max 4, worker 1, source/config/command hashes and cleared inherited provider/LIVE/other-database variables. Cleanup found zero owned database clients or local runtime processes. At prior head `6696a37f8a5718cd0f79054969a0084462d41d47`, all required CI gates pass. The exact-head inventory has 55 successes, one neutral review and three intentional skips. Greptile is 5/5. The complete latest Ponytail verdict confirms the quota finding is fixed; its only remaining note is optional unused MCP proof state, not a behavior defect. All three review threads are resolved. The earlier 643 result remains historical. The repository-wide typecheck, tests, build and browser gates passed in the prior `6696a37f8a5718cd0f79054969a0084462d41d47` CI run. The new `8c912e1ba4fc66397a7341ade59373af8075068f` full CI also passes. They were not run as a full local workspace command in the focused PostgreSQL lease. Local evidence above uses bounded external databases and does not claim embedded-cluster, live-provider or deployed-alarm qualification. ## Risks - An unsupported or uncertain owner state must remain blocked. An optimistic exemption could lose work while compute is stopped. - Changes during reconstruction or inspection must invalidate the proof. A memory-only timer is not durable alarm coverage. - Current authorization, budgets, cancellation, accounting, and resource ownership still apply on dispatch. The proof does not grant execution authority. - The supported writer model uses in-process transaction signals and startup/resume reconstruction. Independent SQL writers and other replicas need an explicit wake contract. - Native, review, coalesced, and taskless retry variants remain conservative until their owner cases qualify. Reviewer quota and hosted alarm qualification remain open. - The shared catalog remains opt-in. Legacy callers and missing owner bindings retain their old blockers. ## Model Used OpenAI `gpt-6.1-sol` was the configured model for the delegated service implementations and reviews. OpenAI Codex, based on GPT-6, performed managing review and integration. Both used repository tools and code execution. The exact served backend snapshots and context-window limits were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id - [x] I have run tests locally and they pass on the final merged head - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green — exact current-head run passes - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups — complete current-head review and all paginated threads inspected - [ ] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
1206373a95 |
fix(environments): close Apex findings on managed plugin environments (#19526)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A managed instance gets its instance-level environments from the `PAPERCLIP_MANAGED_CONFIG` document, and #19508 added plugin environment entries to it > - Greptile's Apex review of #19508 arrived after it merged, with three findings > - A credential inside an array nested in an array passed the secret check and the response redaction > - Recovery of an archived environment listened on one worker handle, which a restart replaces, and its attempts were not tracked through idle checks and shutdown > - This pull request closes all three findings > - The benefit is that credentials never pass the check, an environment comes back after any worker restart, and shutdown never cuts off a recovery write ## Linked Issues or Issue Description Refs #19508 (the Apex findings this pull request fixes) **Subsystem affected** Server environments: the managed-config parser, the environment routes' response floor, the managed environment bootstrap, and the plugin worker manager. **Problem or motivation** Greptile's Apex review of #19508 found three problems. (1) `findSecretLikeConfigKey` and `redactSecretLikeConfigKeys` skip array elements that are arrays, so `{"groups":[[{"apiToken":"..."}]]}` passes the boot check and is echoed to restricted readers. (2) Recovery subscribes to `ready` on one worker handle. A restart such as a settings save replaces that handle, so the new worker's `ready` never reaches the listener, and the environment stays archived. (3) A recovery attempt is fire-and-forget. The worker manager's work token ends when the worker replies, but the database write continues, so an idle check or the database shutdown can overlap it. **Proposed solution** Walk arrays at any depth in both functions. Add a manager-wide `onWorkerReady` subscription and use it for recovery. Run every recovery attempt through a controller that tracks it as idle work and that shutdown stops and awaits before the database closes. **Alternatives considered** Re-subscribing on each new handle after a restart. That needs a hook in every restart path. The manager-wide subscription covers all of them in one place. **Roadmap alignment** This only fixes code that #19508 added or touched. It adds no product surface. ## What Changed - `server/src/services/managed-config.ts`: `findSecretLikeConfigKey` walks arrays at any depth (new `findSecretLikeConfigKeyInValue`). This covers sandbox `config` and plugin `driverConfig`. - `server/src/routes/environments.ts`: `redactSecretLikeConfigKeys` walks arrays at any depth (new `redactSecretLikeConfigValue`). - `server/src/services/plugin-worker-manager.ts`: new optional `onWorkerReady(listener)` on `PluginWorkerManager`. It fires on every worker `ready`, including a replacement handle from a restart. A listener error is logged and does not affect the worker. - `server/src/services/managed-environments.ts`: - Sandbox and plugin recovery subscribe through `subscribeWorkerReady`. It prefers `onWorkerReady` and falls back to the handle when the manager has no such method. - New `createManagedEnvironmentRecovery()` controller. Each attempt runs inside `trackIdleWork` for its whole duration. `stop()` removes every `ready` subscription, refuses new attempts, and awaits unfinished ones. - `server/src/index.ts`: creates the controller, passes it to `applyManagedEnvironments`, and awaits `stop()` during shutdown, after the delivery workers stop and before the application teardown and database close. ## Verification - `cd server && npx vitest run src/__tests__/managed-config.test.ts`: a secret-like key in an array nested in an array refuses boot, for plugin `driverConfig` and sandbox `config`. - `npx vitest run src/__tests__/environment-routes.test.ts`: a nested-array credential is removed from the response, and other nested values stay. - `npx vitest run src/services/managed-environments-plugin-entries.test.ts`: - With `onWorkerReady`, recovery never consults the handle. It ignores another plugin's `ready`, re-ensures on this plugin's `ready`, and then unsubscribes. - An attempt counts in `idleWorkSnapshot()` until its ensure settles. `stop()` unsubscribes, waits for the unfinished attempt, and later `ready` events start nothing. - `npx vitest run src/services/managed-environments.test.ts src/__tests__/managed-plugin-environments.test.ts`: pass. - `pnpm --filter @paperclipai/server typecheck` passes. - I did not run the full `pnpm test:run`, `pnpm -r typecheck` or `pnpm build`. CI covers them. ## Risks - The redaction now also walks nested arrays in sandbox config responses. Only credential-shaped keys are removed; other values are unchanged. - The boot check now refuses a document that hides a secret-like key in a nested array. A valid document has no such key. - `onWorkerReady` is optional, so other `PluginWorkerManager` implementations and test fakes keep working. - Shutdown now waits for an unfinished recovery attempt. An attempt is one worker call plus one short transaction. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (file edits, shell, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
e3fb4bc73a |
fix(native): recover saved listener startup failures (#19479)
## Thinking Path > - Paperclip manages AI agents and their work. > - Native recovery must preserve saved work and respect current task authority. > - A closed local API listener can stop startup before a provider starts. > - #19462 gives new failures a bounded bootstrap retry. > - Older saved failures still have a generic terminal code and a recovery hold. > - This pull request verifies the old producer outcome and uses the existing safe replacement path. > - Eligible tasks continue once. Unknown provider outcomes and user holds remain blocked. ## Linked Issues or Issue Description Refs: #19462 **What happened?** A restart can leave an old listener startup failure in `terminal_failure` with `native_session_interrupted`. Startup recovery does not claim it after #19462. A completed file recovery can also retain its no-replay hold. **Expected behavior** Continue automatically when durable records prove that no provider started, the source stopped, and the current files were saved. Keep the existing attempt budget and normal admission checks. **Steps to reproduce** 1. Close the actual API listener after a fresh native coordinator claim. 2. Persist the old executor's generic failure. Save the workspace and confirm source termination. 3. Restart recovery. The baseline schedules no successor. The repair schedules one successor after all receipts pass. **Paperclip version or commit** Baseline: `0252421d5c`. Old failure producer: `2f35eb1080`. **Deployment mode** Built from source. Local macOS arm64, Node 26.4.0, real embedded PostgreSQL, independent filesystem execution. No external AI service is used by the controlled acceptance. ## What Changed - Recognize the exact saved local-listener guard through one proof predicate shared by discovery and the locked transaction. - Require a fresh first attempt, matching execution bindings, no prior provider lineage, no provider events or tool calls, no checkpoint or admission receipt, and no cancellation. - Reuse current snapshot, exact stop, source block, attempt budget, and successor idempotency checks. - Wait for completed source file-recovery receipts. Retire only that same source's verified preservation hold in the successor transaction. - Bound discovery to 100 candidates per pass, keep completed history quiet, and wake through existing transactional producer notifications. - Retain retries after transient database and final ownership-proof errors; advance the bounded cursor and force a rescan so a failing page cannot hide later work. - Let only the replacement duty discharge a proved-empty notification during drain through its existing read-only readiness check, once per notification generation. Keep real work and concurrent changes outstanding without running replacement during drain. - Wake replacement at canonical local-stop proof writes and the existing dispatch eligibility boundary; these hints grant no new execution authority. - Add database regressions, scheduler quietness checks, real controller handoff coverage, and an opt-in controlled Runner continuation. ## Verification Final production head: `0df41b2fef2644195d1224d36984af3827da3e3b`. Local environment: macOS arm64, Node 26.4.0, TypeScript 7.0.2, Vitest 5.0.3, frozen-lockfile dependencies. Tests execute freshly; `--fsModuleCache` reuses only content-validated module transforms. - Native safe replacement: 62/62 PASS on isolated PostgreSQL 17 (63.18 seconds), including clean per-suite database disposal. All 62 assertions also passed on embedded PostgreSQL 18 beta, but its Darwin cleanup hook timed out; that invocation is not counted as a pass. Docker's disk was full, so the disposable PostgreSQL 17 data used a size-limited RAM mount, with no host mounts. The owned container was removed afterward. - Actual stopped-process producer suite: 31/31 PASS. A real exited child produces the stop receipt, and its replacement notification settles only after the outer transaction commits. - Workspace and separate-process drain regressions: 33/33 PASS (75.90 seconds). - Shared coordinator and execution-control lifecycle checks: 39/39 PASS (16.43 seconds). Pending work, concurrent changes, cold startup, shutdown, and passive drain boundaries remain covered. - Actual suppressed Cloud routine/drain workflow: PASS (34.64 seconds), using the unchanged test and timeouts. Its matching local RED reproduced CI's 462-query, zero-pending-query, one-idle-hold timeout after an admitted write committed during drain (69.46 seconds). - Actual historical controller handoff: PASS (48.40 seconds). A separate controller loads the original executor from `2f35eb1080`, writes its generic listener failure, and exits. Recovery at the final head admits exactly one successor. The real Rust Runner and controlled Codex process complete governed `paperclip_finish`; attempt two commits and the saved file remains intact. Only the child loads the old artifact, and both isolated artifacts are restored afterward. Rust source subtree `77961b70f177ec3c38d11abcaec92fd3d0e96ec0` matches the built binary. No external AI service is used. - Seven corrected existing fixtures pass: transactional stop-proof calls (5 cases) and notification inventory (2 cases). Their earlier CI failures exposed fixture calls outside the documented transaction and an outdated expected queue set. - Server typecheck: PASS. Red/green proofs also cover the original saved failure on baseline `0252421d5c` (zero successors instead of one), held-history quietness, bounded forward scanning through 100 parked rows and concurrent wakes, real snapshot/cleanup producer notifications, actual issue-row lock timeout, and task/history eligibility notifications. An obsolete prior-run question can be skipped to request another proof check; actual skip while the task is paused or the source owns provider tool activity schedules no successor. The failed-page regression starts with 25 persistent verifier errors and a healthy candidate 26. Baseline RED repeats the first page. Repair GREEN advances the healthy candidate, revisits the original failures after their fault clears, creates all 26 successors exactly once, and becomes quiet. Greptile is SUCCESS, 5/5 on the exact final head (2026-11-06 17:09:21 UTC). Final-head CI is GREEN: 56 successful checks, three intentional skips, one neutral result, and no pending or failing checks. [Attempt two and results](https://github.com/paperclipai/paperclip/actions/runs/42258352876/attempts/2) include passing build, typecheck, Rust, native server, package, browser E2E, serialized server, general server/chat tests, and final verification. Five jobs in the first attempt were interrupted together at 17:13:12 UTC; the native-server log records an explicit runner shutdown signal. The single infrastructure retry reran only those five jobs and dependent verification, preserving the 49 successful checks on the unchanged head. No timeout was weakened, and setup failures or interrupted invocations are not counted as passes. Focused commands from the repository root: ```sh ./server/node_modules/.bin/vitest run --project @paperclipai/server server/src/services/remote-workspaces.integration.test.ts server/src/services/native-runtime/native-runner-restart-preparation.integration.test.ts -t 'native safe-replacement decision|reconciles the drained remote backend' --fsModuleCache --reporter=dot ./server/node_modules/.bin/vitest run --project @paperclipai/server server/src/__tests__/execution-control-work.test.ts server/src/__tests__/delivery-work-coordinator.test.ts -t 'execution-control lifecycle|work coordinator' --fsModuleCache --reporter=dot node node_modules/typescript/bin/tsc -p server/tsconfig.json --noEmit --pretty false ``` The historical acceptance sets `PAPERCLIP_TEST_HISTORICAL_LISTENER=1` and uses an original-executor child artifact with a `try/finally` restore. Normal regressions require no Rust binary. ## Risks - Compatibility is deliberately limited to positively proved fresh remote startup failures. Other old ambiguous failures remain held. - Existing provider work, a changed task block, incomplete preservation, or revoked authority must prevent execution. Normal retry admission remains in control. - This PR does not deploy the change or mutate a hosted task. Hosted recovery still requires the repaired revision and all current proof gates. ## Model Used OpenAI GPT-6 through Codex, with reasoning, tool use, and code execution. The exact backend model identifier and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
98ebeefbc9 |
feat(environments): let managed config declare plugin environments (#19508)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A managed instance gets its instance-level environments from the `PAPERCLIP_MANAGED_CONFIG` document > - #19490 added the plugin environment rows, their marker index and write floor, but nothing creates them yet > - The managed config still accepts only one sandbox entry, so a managed instance cannot get a plugin-provided environment without manual setup > - This pull request lets the managed config declare plugin entries beside the sandbox entry, and provisions one locked row for each at boot > - The benefit is that a managed instance gets its plugin environments with no manual setup, and the rows recover by themselves when their plugin restarts ## Linked Issues or Issue Description Refs #19490 (the plugin environment rows this pull request provisions) **Subsystem affected** Server environments: the managed-config parser, the managed environment bootstrap, and the environment routes' write floor. **Problem or motivation** A managed instance cannot get a plugin environment from its managed config. The `environments` section accepts one entry, and that entry always provisions the managed sandbox. Company settings cannot create a plugin environment either. So each managed instance that needs one requires manual setup. **Proposed solution** Let an `environments` entry declare `driver: "plugin"`. Boot provisions one locked row for each plugin entry through the row layer from #19490. A later pull request adds a one-time instance default for one plugin entry, and advertises the new managed-config capability. **Alternatives considered** Manual setup on each instance, and a new control-plane endpoint. Both need a step outside the deploy. The managed config already delivers the sandbox entry. **Roadmap alignment** This extends the existing managed-config `environments` section, which already provisions the managed sandbox. It adds no new product surface. ## What Changed - `server/src/services/managed-config.ts`: an `environments` entry may set `driver` to `"sandbox"` or `"plugin"`. An entry without `driver` is still the sandbox entry, so today's documents parse unchanged. - A plugin entry has `name`, optional `description`, `pluginKey` (the plugin's manifest id), `driverKey` and optional `driverConfig`. - At most one sandbox entry. Names are unique across the section, and `pluginKey`–`driverKey` pairs are unique. Secret-like keys in `driverConfig` refuse startup, as for sandbox `config`. An unknown key or driver still refuses startup, and that includes `instanceDefault` until the follow-up pull request. - Plugin entries parse into a new `pluginEnvironments` list. The `environments` list keeps only the sandbox entry, so its consumers do not change. - `MANAGED_ENVIRONMENT_DRIVERS` lists the accepted drivers. - `server/src/services/managed-environments.ts` (`applyManagedEnvironments`): - Boot refuses to start unless an auto-installed catalog entry installs each plugin entry's `pluginKey`. `plugins.autoInstall` holds catalog keys, not manifest ids, so the parser cannot check this. Boot uses the manifest ids `createApp` already resolves from the bundled and distribution catalogs (`app.locals.managedBundledPluginKeys`, passed in by `server/src/index.ts`). Without that list it resolves the catalog itself. - Plugin entries run after the sandbox entry and reuse its readiness handling. Boot awaits `pluginsReady`, then requires the plugin `ready`, declaring the entry's driver, and running a worker. Otherwise boot archives the row and re-ensures it on the worker's `ready`. - Recovery re-ensures on every worker `ready` until one ensure succeeds or the plugin rejects the config (HTTP 422). - If the ensure fails because the worker died, boot archives the row and waits for `ready`. If the worker is still running (for example the call timed out), boot keeps the row as it is, retries once at once and on the next `ready`, and otherwise at the next boot. No timer is added. - Removing an entry never archives its row. - A plugin-only document does not conflict with `PAPERCLIP_EXECUTION_MODE`, because plugin rows do not use the sandbox slot. - Plugin entries never change the instance default. The sandbox stamp still fills only an unset or local default, so it never replaces a plugin default. - `server/src/routes/environments.ts`: `isPlatformSlotEnvironment` treats a marked plugin row as a live slot while the document declares its entry. So the marker-clear escape hatch is refused then, and allowed again after the entry leaves the document. - `doc/DEVELOPING.md`: new "Managed environments" section. ## Verification - `cd server && npx vitest run src/__tests__/managed-config.test.ts`: old documents parse unchanged. Sandbox plus plugin parses. Each of these refuses startup: two sandbox entries, `driver: null`, duplicate names, a duplicate pair, a secret-like key, an unknown key (including `instanceDefault`), an unknown driver, a malformed `driverKey` or `driverConfig`. - `npx vitest run src/services/managed-environments-plugin-entries.test.ts` (the sandbox-entry tests stay in `managed-environments.test.ts`; shared fixtures are in `managed-environments.test-support.ts`): - A plugin entry is ensured with the validator's worker manager, after the sandbox entry. - Boot refuses a plugin entry whose `pluginKey` no auto-installed catalog entry installs, both from the passed list and from its own catalog lookup. - A plugin that is down archives the row, and worker recovery re-ensures it. A missing plugin, a plugin without the driver, or a worker manager without `call` counts as unavailable. - Recovery stays armed after a failed attempt and stops after one success or a permanent config rejection (422). A `ready` during an attempt queues one more. - A worker that dies during the boot ensure archives the row and arms recovery. A failure while the worker keeps running leaves the row as it is and retries. - Removing an entry does not archive. Plugin entries and the sandbox stamp never change a plugin default. - An ensure failure is counted, not thrown. - `npx vitest run src/__tests__/managed-plugin-environments.test.ts` (embedded Postgres, real `applyManagedEnvironments`): the first boot creates the row and the second changes nothing. A plugin that is down archives the row, and recovery brings it back. Removing the entry does not archive the row. The instance default stays unset. The row tests from #19490 still pass. - `npx vitest run src/__tests__/environment-routes.test.ts`: the marker clear is refused while the document declares the entry, and allowed after it leaves. - `npx vitest run src/__tests__/instance-settings-service.test.ts src/__tests__/instance-settings-cloud-defaults.test.ts`: pass with the new `pluginEnvironments` field. - `pnpm --filter @paperclipai/server typecheck` passes. - I did not run the full `pnpm test:run`, `pnpm -r typecheck` or `pnpm build`. CI covers them. ## Risks - No schema change. - A managed-config document with a plugin entry now provisions a row at boot. A document without one behaves as before. - Adoption renames an unmarked row for the same plugin and driver to the declared name and resets its driver config to the declared one. The row keeps its id, agents and other metadata, and it is then locked like the managed sandbox. - Boot now refuses to start when a plugin entry names a plugin that no auto-installed catalog entry provides. This is the same fail-closed posture as an unknown sandbox provider. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (file edits, shell, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
8c578668b8 |
fix(heartbeat): start new queued runs promptly after restart and name the wait (#19431)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The heartbeat queue starts queued agent turns. Restarts and deploys
happen often.
> - After a restart, new queued turns waited minutes before they
started. The task showed only "Queued", with no reason.
> - The cause is startup order. The queued-run dispatch worker registers
only after many unrelated first passes finish one after another: tool
reviews, delivery workers, the terminal workspace sweep, native restart
recovery, and residual and issue recovery.
> - The server binds its listener before this recovery, and
`/api/health` answers during it. A deployment can route traffic to the
new server for minutes before queued work starts.
> - This pull request starts queued turns that never executed as soon as
plugin workers and adapters are loaded. It also names the wait while the
server starts, recovers, or shuts down.
> - The benefit is that tasks resume quickly after a restart and show a
visible reason while they wait.
## Linked Issues or Issue Description
**What happened?**
On a hosted instance, the server restarted twice for deploys. Both
times, queued runs on idle agents started about 3.5 to 4 minutes after
the restart. Before the restart, the runs also waited about 5 minutes
while the deploy drained the old server. During the wait after the
restart, the execution presentation showed `phase: queued`, `label:
"Queued"`, `cause: null`.
**Expected behavior**
Queued work resumes automatically and quickly after a restart. While it
waits, the task shows a reason.
**Steps to reproduce**
1. Start a task drain (`POST /api/instance/task-drain`). Assign issues
to idle agents. The runs stay queued.
2. Stop the server (SIGTERM) and start it again.
3. Make any startup first pass slow (a large instance does this
naturally). Queued runs start only after every earlier startup pass
finishes.
**Paperclip version or commit**
master at
|
||
|
|
5c185db77c |
fix(tree-control): resume work a subtree pause interrupted (#19394)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Task tree control lets the board pause a task tree and then resume
it.
> - A pause stops the runs that are active in the tree. The release must
then continue the work that was pending.
> - A child task whose run was active at pause time did not start again
after release. The child became blocked and asked for a user message.
> - A pause is an operator hold, not a failure. It must not strand work,
and it must not require reconciliation for a run that never started.
> - This pull request re-dispatches interrupted work one time on
release, and it settles a legacy Stop that never entered the adapter.
> - The benefit is that pause and resume keep a task tree moving without
manual recovery, and real safety gates stay in force.
## Linked Issues or Issue Description
**What happened?**
A parent task delegated two children. One child's run was active (in
host workspace preparation, before the adapter started) when the board
paused the parent's tree. The pause cancelled that run. After the board
released the hold, no new run started for that child. The child became
`blocked` with the execution blocker cause
`legacy_execution_requires_reconciliation`. Later the next action became
"Send a message to continue." A sibling child that had no active run at
pause time resumed normally.
**Expected behavior**
A run that a subtree pause cancels is an operator hold. On release, the
pending work starts again one time, automatically. It does not go
through execution reconciliation when the run never entered its adapter.
It does not wait for a user message.
**Steps to reproduce**
1. Create a parent task and a child task that is assigned to an agent.
2. Wake the child. While its run is in host preparation (or queued),
call `POST /api/issues/:parentId/tree-holds` with `{ "mode": "pause" }`.
3. Call `POST /api/issues/:parentId/tree-holds/:holdId/release` without
`wakeAgents`.
4. Before this change, no new run starts, and `getExecutionBlocker`
reports `legacy_execution_requires_reconciliation` for the child.
**Root cause**
- (a) The pause records each interrupted run in the hold member snapshot
(`activeRunId`). Nothing used that record on release.
- (b) Release woke tasks only when `metadata.wakeAgents === true`.
Without it, release relied on stranded-issue recovery. That recovery
stands down when the latest run was cancelled by an operator
(`isOperatorCancelledRun`). So the interrupted child had no owner.
- (c) The pause Stop of a legacy run in preparation is written directly
as `cancelled`. Legacy terminalization classifies a cancelled run with
no bootstrap evidence as an unknown outcome. It records
`legacy_execution_requires_reconciliation`, although the gated dispatch
never entered the adapter.
- Native runs: the verified native Stop does not create the
reconciliation hold, but gap (b) applies to native runs too. The release
change is runtime-independent.
## What Changed
- `issueTreeControlService.findOwedPauseResume` decides if a task is
owed one resume. The task's latest run must be an operator Stop that a
pause hold member snapshot names, recorded while that hold was active.
The task must still be executable by the same agent, and no pause may
still cover it. The obligation belongs to the run, so overlapping pauses
keep it until the last covering release.
- New `services/issue-tree-resume.ts` re-dispatches owed work with wake
reason `issue_tree_resumed` (`holdId` and `interruptedRunId` in the
context). A compare-and-set claim (`treeHoldResume`) on the interrupted
run makes the resume happen at most once. A failed wake returns the
claim.
- Pause release and `mode: "resume"` call it for every member of the
released hold, also without `wakeAgents`. When `wakeAgents` is true, the
existing loop wakes every executable member.
- A task that still has an execution blocker is not replayed. The
release writes `issue.tree_resume_wake_skipped`, the blocker stays
visible, and the obligation stays unclaimed.
- `releaseHold` updates only an `active` hold. A repeated release gets
`409`.
- `settleCancelledLegacyPreparation`: when the executor that owned a
cancelled legacy run proves that dispatch never entered the adapter, it
records bootstrap evidence and resolves that run's
`legacy-execution:<runId>` hold as `false_positive`. It also corrects an
automatic no-replay disposition and restores a `blocked` status only
when that disposition made the change. Then it re-dispatches owed work,
for a release that happened while the Stop was unwinding. It requires a
requested Stop with no existing recovery receipt, and no process,
session, native finalization, unreleased lease, or execution-start
event.
- `settleUnrecoverableExecutions` waits while a legacy executor still
holds its controller lease, and records a `legacyCancellationBlock`
receipt when it blocks a task for a cancelled legacy run.
- Docs: `doc/execution-semantics.md` and `doc/composer-stop.md` describe
the resume rule.
## Verification
- New regression
`server/src/__tests__/issue-tree-pause-resume-run.test.ts` uses embedded
Postgres, the real heartbeat, and the real tree-control routes:
- A run paused during preparation: nothing runs during the hold, there
is no blocker, and release creates exactly one `issue_tree_resumed` run
that succeeds. A second release returns `409` and creates no run.
- Release before the stopped preparation finishes unwinding (with the
periodic disposition run in between): exactly one resumed run after the
executor settles.
- An automatic no-replay disposition that ran first: the blocker clears,
the status returns to `in_progress`, and release resumes once.
- Overlapping child and parent pauses: no resume while the parent pause
remains, then exactly one resume after it is released.
- A queued run cancelled by the pause: release creates exactly one
resumed run.
- A task reassigned to a user during the hold: no resume.
- A mid-turn legacy Stop with unknown outcome: no replay, the
reconciliation blocker remains, and `issue.tree_resume_wake_skipped` is
logged.
- Red before each fix: the first 4 cases failed on master; the 3 review
cases failed on the first PR commit. All 7 pass now.
- Also passed locally: `issue-tree-control-routes`,
`issue-tree-control-service`, `heartbeat-workspace-busy`,
`heartbeat-undispatched-execution-wait`, `execution-control-work`,
`legacy-workspace-restore-recovery`, `legacy-continuation-authority`,
`task-retry`, `legacy-execution-recovery`, `agent-conversations`,
`heartbeat-retry-scheduling`, `satisfied-connection-intents`.
- Not verified here: a live staging pause/resume. Re-check on staging
after deploy.
## Risks
- Behavior change: release now starts the work that the pause
interrupted, also without `wakeAgents`. Before, that work stayed idle.
Other idle tasks still wake only with `wakeAgents`.
- The settlement trusts the executor's adapter-entry flag. The flag is
set only inside the atomic dispatch gate, which refuses a cancelled run.
The settlement also requires no process, session, lease, or execution
event.
- A native run whose Stop has not finalized at release time is not a
target. Stranded recovery and explicit wake still apply.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edit, test execution).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
5f1b59ba2c |
fix(db): preserve binary data in checkpoint backups (#18650)
Reviewed and verified at
|
||
|
|
27d35314be |
feat: connect experimental Cursor Cloud to native Runner (#18069)
## Thinking Path > - Paperclip helps people manage AI agents at work. > - Native Runner owns execution and task authority. > - Operators need an explicit native Cursor Cloud choice. > - Tools/files need company, task, session and run scope. > - This connects setup and lifecycle to Runner Core. > - Legacy agents retain their selected runtime/history. ## Linked Issues or Issue Description Depends on #18068. Managed-only API placement and API-owned Stop containment are in #18310. Refs #5664, #11803, #17299, and #17471. No duplicate native integration was found. ## What Changed - Add explicit native setup, Test, saved settings and dispatch with no-prompt account/catalog/callback checks. - Scope MCP to company, agent, task, session and run through authenticated HTTPS. Register verified artifacts/work products; preserve unpriced accounting, completion and budgets. - Connect Question/Plan continuation, cold handoff and admitted fresh sessions; recheck waiting admission under the task lock. - Present audited Stop intent while preserving physical containment proof; bound typed Stop metadata. Preserve title edits through preference/navigation readiness. - Select one live/history representation per run to prevent duplicate settled cards. Integrate upstream task/read-state/retry/schema contracts and repair two test fixtures. ## Verification Core #18068 merged normally as `3910352c923a`. Product head `5a60f541c982` has passing repository CI/build/browser checks, including required `ci / verify` and `ci / e2e`, plus a fresh exact-head Greptile 5/5 review with zero unresolved threads. Two non-required external app checks remain pending: Paperclip Review is queued waiting for its assigned agent; Superagent Security Scan is reviewing the diff. These are not claimed completed. The normal merge endpoint enforces the repository rules; no policy or status override is used. The dependent Top PR is addressing its fresh deferred-Stop review finding. Product `5a60f541c982` passed 270 UI and 20 selected Cloud/native/disk-full tests, UI/server typechecks, tokens and maps. Top’s clean normal merge passed tokens/maps/diff checks; its matrix is pending. Earlier receipts retain Core’s 47 Rust, 4 compiled-native retirement and 19 settlement controls, normal Runner build, and runtime 9af build/recursive typecheck. Reproduced loading/shutdown CI fixture failures passed 7/7 browser and 121/121 unit tests, including a broken-unsubscribe negative control. No timeout/oracle was relaxed. The actual-parent restack adopts 75 mainline paths, including schema/permission and generic runtime changes, beyond the prior two fixture repairs and client projection fix. The full difference from hosted `9af1a5de42a2` is therefore larger than four files; backend source also differs. The merged config-read checks match Core391, and the Cursor placement/runtime/relay/selection modules match frozen9af; selected affected regressions pass. Paid proof keeps its actual 9af source/image identity. The separately tested live/history card fix is not claimed deployed on that host. On 9af, original saved settings survived reopening and fresh no-prompt Test authenticated catalog/model/binding; no new Save is claimed. Fresh Linux admission passed. With `claude-sonnet-4-5`, Artifact downloaded exact 27 bytes and matched 121 canonical identities in complete S3-served history. First Stop’s strict quiet FAILED on a 10.373-second gap despite independent cancellation/no late artifact. One targeted unchanged-sampler retry retained all 24 samples: 35.076 seconds quiet / maximum gap 4.584 seconds, 67 ordered canonical identities, and separate CANCELLED/artifact-absent status after the original 180-second deadline. The pre-reload duplicate settled card is disclosed. Exact-ID cleanup for the three new provider agents is awaiting authorization, so full grades and the fresh Artifact’s post-cleanup download remain blocked. A historical deleted-provider artifact was newly downloaded on 9af; original F226 Question/Plan/reconnect and cleanup proof keep their identities. Retained failures: stale fixture 132/133; E8 48/135 with 87 reset-hook failures; Mac 17 SQL failures/skips; earlier title/startup/custody/Stop/deletion failures with unknown causes/bounded recoveries; `dea023` Stop’s warning/50 duplicate identities; pre-repair ACK and original Core CI failures. WSP’s 251-assertion, 7/8-suite attempt failed physical PostgreSQL teardown; the unchanged wrapper rerun passed 128. Retry alone establishes no cause or full local-suite pass. All paid attempts remain recorded; vendor dollar cost is UNKNOWN. ## Risks Experimental: Cursor independently acknowledges neither the effective model nor instruction digest before the first prompt. Explicit values are submitted and checked through behavior; unpriced provider dollar cost stays UNKNOWN. Callbacks need authenticated reachable HTTPS. Scope, revocation, budgets, completion and cleanup remain authoritative. Operator Stop intent is separate from physical containment. Saved legacy agents retain their selected runtime. Prior shared title-save/deletion failures remain disclosed. ## Model Used OpenAI Codex, GPT-6 family, assisted with reasoning, repository tools, code execution, browser verification, and delegated code review. The serving model’s exact API ID and context window are not exposed. No value is inferred from the model catalog. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used and the limits of available model metadata - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have linked public issues or described the issue in this PR - [x] I have not referenced internal or instance-local Paperclip issues or links - [x] My branch describes the change and contains no internal ticket ID - [x] I have run the scoped local tests reported above; pending gates are stated separately - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation - [x] I have considered and documented the risks - [x] All required CI gates pass on the current head - [x] Fresh Greptile is 5/5 with no unresolved findings on the current head - [x] I will address all reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
a9b18e1cbe |
fix(runner): preserve cleanup ownership across startup failures (#18658)
## Thinking Path > - Paperclip controls which run owns an agent task. > - A plugin can continue accepted work after Core stops waiting. > - A late task-creation reply can therefore arrive after startup cancellation. > - Core must keep that work owned until its resource identity and cleanup outcome are known. > - This PR supplies the cleanup and recovery support before shared startup is enabled. > - The dependent activation PR can then reuse transport data within a bounded startup context. ## Linked Issues or Issue Description Refs #18405. This is the independently mergeable prerequisite for that stacked activation PR. **What existing behavior does this improve?** Cleanup of explicitly recorded startup work that outlives its caller. **Current behavior** Caller timeout, worker exit, or failed cleanup-intent persistence can leave work without a complete recovery path. **Proposed behavior** Retain the cleanup obligation, use existing Stop and recovery queues, and release ownership only after the exact task and accepted work have stopped. Ordinary calls keep their existing startup payloads and budgets. This PR does not create startup contexts or advertise transport reuse. **Reason and benefit** Separate the failure-handling contract from activation so each can be reviewed on its own. This support PR has **no standalone TTFT speedup**. ## What Changed - Add the startup cleanup marker, exact ownership checks, verified completion, and acknowledgement helpers. - Connect recorded cleanup obligations to existing Stop, deadline reconciliation, restart recovery, and retry scheduling. - Retain typed cleanup responsibility when a database write fails. Keep the nonterminal run from being treated as ordinary completed cleanup. - Observe late acquisition results and save resource identities before retiring them. Retain real host writes and received results across worker exit or persistence failure. - Queue abandoned replacement tasks through the existing creation journal and cleanup machinery. Require the replacement's own Stop proof. - Accept an explicit startup context in the internal bridge for lifetime tracking. Automatic context creation, marker production, capability advertisement, transport capture, and executor activation belong to #18405. - Add PostgreSQL and worker IPC regressions, including legacy payload assertions. ## Verification Initial split source `8fb482d8be9e70101b0cfdd0830f41cf5b8e3cf4` had tree `95508bd1859e991670cad8afd4de8d2167009ac5`. **190 distinct targeted tests pass**, along with server typechecking. The selected checks cover real PostgreSQL cleanup/ownership, worker IPC, the bridge and creation journal, ordinary/distributed Stop, restart and deadline recovery, idle-sleep behavior, and the unchanged legacy warm-task suite. Independent recovery and worker reviews found no blocking issue. The first correction, `b6342d42042601b376da38dc8e1dba9ca80e9539`, fixes the replacement cleanup finding with the existing startup-cleanup marker, scoped to contextual abandoned task creations. The replacement was never prepared, so preparation-stop proof is not required. No dispatch is fabricated and no normal-startup query is added. The actual queue/sweep/driver/receipt/cancellation regression fails before the fix and passes after it: void, running or foreign-task status retains ownership; exact-task stopped status permits completion even when Release returns void. **64 affected targeted cases and server typechecking pass**. Independent review found no blocker. Current source is `f53af72af76f263096c3c2e788506e6faef9474f`. It fixes both recovery findings from the prior head's Apex 3/5: orphan recovery now preserves pending or completed-but-not-cancelled cleanup through its existing final-write fence, and exact late identities can reach verified cleanup after reconciliation or restart. The regression selection went from four failing assertions to six passing cases. Affected selections cover 217 distinct cases; the exact final six-case regression and server typecheck pass. Independent reviewers found no blocker in either fix. Current-head standard review is 5/5 and all CI checks pass (54 successes, three expected skips, plus Snyk). Apex also completed at **5/5** on this exact head, with zero unresolved threads. The requested independent availability audit is complete on this head. It found an activation-dependent operability gap: an acquisition response that is never received can leave an unknown creation identity with no supported task inspection/reconciliation path, while cleanup keeps the run owner reserved. This does not occur in ordinary lower-only calls. The contract and recovery policy need follow-up before activation; the external review scores do not resolve that availability tradeoff. The split preserves the accepted implementation at `21db7b3851` after its clean merge with frozen master `946735d03a55c1fb4cc5434d6bf1d87742daa9df`. The initial dependent activation PR restored exact aggregate tree `f68c6895f87c7b93ae03e6324d955d2659a99dce`. Both subsequent cleanup corrections are integrated into upper head `27978f294a02da3992585c7f4daa73cd8fcf99f8`; its current aggregate tree is `a0b66c33f5e22745f034f45d24d213f5c210cf16`. Lower-only payload tests cover the absence of capability advertisement and transport projection. Only targeted local tests and typechecking are used. Full regression coverage runs in GitHub CI. No deployment or ROM qualification is part of this change. The lower audit covered connected worker, creation-journal, Stop, restart, deadline and orphan-recovery paths. A new isolated real-PostgreSQL characterization reproduced the unknown-ID hold through actual recovery, cleanup verification and run-control cancellation. It models a lost acquisition response; it is not a live provider or complete IPC test. No source edits were made by the audit. The upper/composition audit remains required after its review cycle. ## Risks - Recording a replacement result now commits with the existing cleanup notification. This conditional replacement path adds transaction/notification responses; common retained startup is unchanged. - This adds cleanup and recovery state handling. A missing receipt or permanently inaccessible old backend can keep ownership held indefinitely. Temporary outages retry; an ordinary retry is not evidence that a deleted or replaced backend stopped. - A caller timeout does not undo work already accepted by a plugin. Unknown resource creation remains unresolved until its identity/outcome is established. Current task creation has no supported inspect-by-operation or operator reconciliation API for a permanently lost identity; repeated Stop alone cannot resolve it. - The support code understands records produced by the dependent activation PR. It does not enable the shared startup policy by itself. - Existing legacy callers must retain their behavior; context-only automatic replacement cleanup remains explicitly scoped. ## Model Used OpenAI Codex, GPT-6 family. The exact model ID and context window are not exposed in this session. Tool-assisted implementation, targeted testing, and independent subagent review. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used, with available version and capability details - [x] I have checked ROADMAP.md and confirmed this is the requested split of existing work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue with the required fields - [x] I have not referenced internal issues or private links - [x] My branch name describes the change and contains no internal ticket ID - [x] Targeted local tests pass on this source - [x] I have added or updated relevant tests - [x] I have updated relevant documentation - [x] I have documented the risks - [x] All current-head GitHub CI gates are green - [x] Current-head standard Greptile and Apex are 5/5 with no unresolved findings - [x] I will address all review comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
73cd8eb9a5 |
fix(server): deliver answers promptly during web handoff (#18530)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - Its web servers can overlap while Railway replaces one deployment.
> - An answer can reach the replacement while the original server owns
the native execution.
> - A competing delivery claim can make the owner miss the first
notification.
> - The normal retry interval then delays the answer by 30 seconds.
> - This pull request gives pending answers a one-second retry in
ephemeral mode and verifies delivery between separate processes.
## Linked Issues or Issue Description
Refs #18518 and #18526. This closes a measured native-question delivery
delay during control-plane web deployment overlap.
## What Changed
- Register question delivery through its owning service. Use a
one-second retry for ephemeral servers and retain the configured cadence
for ordinary servers.
- Keep the existing durable claims, response identity, authorization and
continuation behavior.
- Add a two-process PostgreSQL regression for a claim held by the
non-owning server. Verify one command, settlement, idle behavior and
reactivation by another committed answer.
- Keep the assembled startup fixture on the real registration function.
Update the deployment plan with current evidence and remaining
acceptance work.
## Verification
- The same local two-process race took 30.157 seconds before the fix and
1.197 seconds after it. The command sink is simulated; PostgreSQL,
notification transport, interaction response, delivery claim and worker
are real.
- The cross-process regression and assembled startup suite passed 122
tests. No further sweeps or pending inspections occur after settlement.
A second answer wakes the owner through PostgreSQL without a test-owned
sweep.
- Server TypeScript compilation passed. All PR CI gates passed on
|
||
|
|
006d5691a6 |
feat(server): support volume-free web server handoff (#18518)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Its web server also owns controller admission and user commands. > - A rolling deployment starts a replacement before the old server exits. > - Shared database state alone does not preserve local agent files, browser chunks, or in-process commands. > - This pull request adds an opt-in ephemeral server contract and a local overlap test. > - The benefit is a web server that can relinquish traffic and local files without losing acknowledged work. ## Linked Issues or Issue Description Refs #18403, #18327, #18335, #18341. **Current behavior** The control-plane web server still has local agent-file authority and local command targets. A new container can miss saved bytes or send a command to the wrong process. An existing browser tab also needs chunks from its original build. **Proposed behavior** Eligible servers share only PostgreSQL, object storage and stable secrets. A replacement waits for readiness. The old server drains admitted requests, keeps owner commands available through handoff, and releases controller leases. Explicit maintenance adopts existing files before the volume is removed. **Reason and benefit** This is the application prerequisite for rolling app-only deployments. Schema migrations and permanent multi-replica operation are outside this change. ## What Changed - Store agent-file manifests in PostgreSQL and immutable bytes in S3. Treat local roots as disposable projections. Preserve existing permission checks and edit conflicts. - Add an explicit migration, eligibility and object-retention command to the production runtime bundle. - Add `/api/ready`, admitted-request drain, browser restart signaling and retained immutable UI chunks. - Carry transaction-aware work hints across controllers during handoff and active execution. Keep listeners closed when idle. - Deliver durable Stop and runtime responses to the execution owner. Recheck request lifecycle and access before response dispatch. - Reject local-only configuration in ephemeral mode. Keep the feature disabled for existing installations by default. - Add an isolated PostgreSQL/S3/proxy/browser deployment harness with independent server homes and retained failure evidence. ## Verification - All CI checks pass on `ce17b95019d3cf1cabd357d19b6d37f96782cc5c`: workspace typecheck, build, Runner checks, server/general tests and browser shards. Greptile reviewed the runtime changes at 5/5 with all findings resolved; the final commit updates maps and the plan. - Local targeted checks passed: coordinator 27; Stop/recovery 33; PostgreSQL command delivery 6; route/workspace regressions 8; bootstrap Stop 1; restart recovery 75. PostgreSQL command tests simulate the provider edge. - Local browser proof passed 91 reads and 19 saves; reconnect took 1,013ms. The tab opened Audit from S3 after deletion of A's whole home/build. This uses compiled workspace code and local-trusted auth, not released containers or live agents. - Local workspace typecheck/build passed; current server and runtime bundle passed. The earlier broad local suite was stopped after three hours and is not a passing receipt. Its retained failures and isolated reruns are documented in the plan. - Stacked #18526 adds stricter graceful-exit proof, an unready replacement control, live Runner qualification and a Stop handoff correction. Hosted enablement remains gated on those results and Cloud #1096. No hosted stack has been enabled. ## Risks - Apply migration 0422 before enabling the feature. Existing data must pass each owner's migration and eligibility checks. Do not discard a volume based on a single migration page. - Durable manifests remain authoritative after the opt-in is disabled. Rolling back to a binary that ignores them is unsupported. - Cold directory restoration and complete shutdown timing still need qualification before enabling a hosted stack. - Retained UI assets must cover the supported browser and rollback window. Cleanup is explicit; no new idle maintenance polling is introduced. - This change does not claim production zero downtime or authorize a fleet-wide enablement. ## Model Used OpenAI Codex, GPT-6 family. The exact serving model ID and context-window limit are not exposed in this session. Used reasoning, code editing, shell execution and browser automation. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
681f7b7726 |
perf(server): retire aggregate maintenance polling (#18425)
Reviewed and verified at
|
||
|
|
9ca9af544a |
Drive remaining live run and resource recovery from durable work (#18419)
Reviewed and verified at
|
||
|
|
eb12cfc96b |
perf(github): activate connection work from events and deadlines (#18395)
Reviewed and verified at
|
||
|
|
5e4cc01073 |
feat(workspaces): keep remote files off the webserver disk (#18403)
Keep native sandbox workspace files independent of the webserver disk. Use encrypted saved versions and Postgres ownership so a replacement server can recover files and continue work. Enable the path by default for new eligible workspaces when durable storage is configured. Preserve explicit opt-outs and existing host workspaces. Include safe snapshot repair, Files access, import, migration and retention. Bound saved-Files resources, preserve imported Git recovery, reject unfinished Git operations, publish immutable concurrent captures, and verify permanent object-version deletion. Validated 471 targeted local controls, build/typechecks, all 40 CI partitions, fresh Greptile 5/5 and security review. The PR records original failed attempts and scoped retries. Co-Authored-By: Paperclip <noreply@paperclip.ing> |
||
|
|
75fa37f30f |
fix(server): expire native backups from retention deadlines (#18380)
Reviewed and verified at
|
||
|
|
453f5fbb78 |
perf(external-objects): schedule scoped refresh obligations (#18384)
Reviewed and verified at
|
||
|
|
c32ecf6657 |
fix(server): schedule only outstanding merge confirmations (#18382)
Reviewed and verified at
|
||
|
|
9a3f395737 |
fix(server): settle routine history without idle scans (#18399)
Reviewed and verified at
|