mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-19 16:29:42 +01:00
8a295f944d280c5556dedba9a2fa35d2be942d21
458
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
651749bdb4 |
feat(connections): AWS connector with credentials minted per connection (#22195)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents use outside services through connections, and the control plane holds each connection's credentials > - Companies want agents to "do things on AWS" with exactly the permissions they choose, without Paperclip storing long-lived AWS keys or rebuilding the AWS API > - AWS hosts an MCP server that runs any AWS API call in its own sandbox, and it accepts a one-hour bearer token that can be minted from IAM credentials > - This pull request adds an AWS connection that mints that token for each connection at call time. The role method proves the connection with the company's workload identity issuer (#22083), so no AWS key is stored > - The benefit is that each AWS connection carries its own role and permissions, AWS enforces them, and agents never see any AWS credential ## Linked Issues or Issue Description Refs #22083 (workload identity issuer) and #22191 (spike findings). **Subsystem affected** Connections (tool access), tool gateway, shared app definitions, instance settings. **Problem or motivation** Paperclip has no way to connect AWS. AWS's hosted MCP server needs a bearer token from AWS Sign-in, minted from IAM credentials. No existing credential path mints a header at call time from a role or key. **Proposed solution** Methods can declare `credentialMinter: "aws_mcp"`. The gateway and connection health checks mint the `Authorization` header at call time, and they share one in-memory cache per connection and grant (about 55 minutes). There are two methods: - Role (`AssumeRoleWithWebIdentity` with a Paperclip-signed connection token). - Access key (the key directly, `GetFederationToken`, or `AssumeRole`). Read-only connections add `ReadOnlyAccess` and `AWSMCPSignInOAuthAccessPolicy` as session policies, so AWS refuses changes. The feature is behind the `enableAwsConnector` experimental setting. **Alternatives considered** - Run AWS's `mcp-proxy-for-aws` process on the control plane. Rejected: it adds a Python runtime and processes, and gives nothing the token exchange does not. - Put credentials in the agent sandbox and use the AWS CLI. Deferred: there is no per-call governance, and every sandbox would need the CLI. - Add a new `credentialSource`. Rejected: it needs a migration, and the `vercel_connect` database rules do not fit stored access keys. ## What Changed - `packages/shared`: - `credentialMinter` on connection methods. The validator requires `mcp_remote` and `api_key`, and does not allow `keyPlacement` or another credential source. - The `aws` app definition (two methods, reviewed tools, `mcpSessionRequired`, experimental key). - The `enableAwsConnector` setting in the catalog, types and validators. - `aws` added to the connectable apps. - `server/src/services/connectors/aws/mcp-credentials.ts`: - Settings parsing and checks. The URL must be AWS's endpoint for the saved region, and the role ARN must be valid. - The minter: role or key session, read-only session policies, Sign-in exchange, cache keyed on settings, key references and status, in-flight de-duplication, eviction. - A plain-language error mapping that never shows AWS's own text, because that text can contain account IDs and ARNs. - Gateway (`resolveCredentialHeadersUnrecorded`), credential access (`resolveCredentialHeaders`), catalog discovery and health-failure mapping: - The AWS branch. - After a 401, the gateway evicts the token and mints again, then requires an explicit retry (no replay). Discovery lists again. - Setup: - Minted methods may save a draft without contacting the provider. The role must trust a connection that already exists. - Saving an AWS role draft publishes the company's issuer, so AWS can validate it when the customer registers it. - `workload-identity.ts`: a draft connection can get a token, so setup can test it. Disabled, archived or switched-off connections still get none. - `server/package.json`: `@aws-sdk/client-sts` and `@aws-sdk/client-signin`. The lockfile is left to the refresh workflow. - Artwork: AWS light and dark marks from LobeHub's MIT icon set (already licensed in the repo). The manifest entry passes the brand checks. - `scripts/ingest-app-definitions.mjs`: minted methods are exempt from the `keyPlacement` requirement. The override source is `scripts/app-definition-overrides/aws.json`. Permission reviews are added. - UI: the experimental settings toggle, in alphabetical position. - Docs: `doc/connections/AWS.md`, the connections index, the issuer API doc (draft rule), and the plan's phase table. ## Verification - `server/src/services/connectors/aws/mcp-credentials.test.ts` (7 tests) covers these items: - Both methods and the read-only policies. - Caching, revision, forced refresh, expiry and eviction. - Concurrent de-duplication. - The error mapping, with no AWS text leaking. - Settings validation, including the URL and region check and the rule that an unknown access level means read only. - `server/src/__tests__/aws-mcp-connector.test.ts` (3 tests, real setup, catalog and gateway code with a fake AWS and a session-requiring fake MCP server) covers these items: - The setting gates setup. - The access key is never sent as a header or stored in config. - A cached token is reused. - After a 401, a fresh token is minted and the call is not replayed. - Role draft-then-resume publishes the issuer. - The signed token has the right `iss`, `sub` and `aud`. - An AWS refusal shows a plain health message without the account ID. - Red/green: - Removing the gateway hook fails the access-key test. - Turning off `mcpSessionRequired` fails both integration tests. - Removing draft support for minted methods fails both integration tests. - Restoring each change passes again. - Nearby suites: `tool-gateway-service`, `tool-access-service`, `remote-mcp-connectors`, `instance-settings-service`, `workload-identity`, plus the new files. 775 of 775 pass. - Shared `app-definitions.test.ts` passes (42), including new `credentialMinter` schema tests. UI `InstanceExperimentalSettings` (63) and the brand asset tests pass. `check-app-brand-assets` validates 112 identities. - Typecheck is clean for `packages/shared`, `server` and `ui`. The only UI errors are in Storybook's generated catalog files, which this worktree has not built. - **Real AWS** (Paperclip dev account, integrated code path, issuer behind a temporary public tunnel): - Access key, read only: 8 tools listed. `run_script` `ListBuckets` succeeded in 8.9 s on a cold sandbox. AWS refused `CreateBucket` ("no session policy allows"). - Role: the draft was saved without contacting AWS, and the issuer was published. After an IAM provider and a role trusting only that connection were created, resuming setup listed 8 tools. `ListBuckets` succeeded in 12.8 s. The role's own permissions refused `CreateBucket`. - This run found two bugs the fakes did not: AWS needs MCP sessions for calls, and the role method needs draft-first setup with the issuer published. Both are fixed and both now have tests. - All IAM users, keys, roles, providers and buckets were deleted, and a listing confirmed it. - Not run: Paperclip Cloud staging (the setting is off by default), and `AssumeRole` from an access key with a role ARN against real AWS (unit tests only). ## Risks - The feature is off by default, behind `enableAwsConnector`. No migration. - New dependencies `@aws-sdk/client-sts` and `@aws-sdk/client-signin` are in the existing `@aws-sdk` family (the server already uses `client-s3`). - Minted tokens are held only in process memory. A restart mints again. A cached token can outlive a deactivated key by up to an hour, unless AWS returns 401, which evicts it. - `aws___run_script` is classified as a write, so the normal approval and ask-first policy applies. Read-only connections rely on AWS session policies. - The guided setup screen and the CloudFormation template are not in this PR. Setup uses the generic form and the documented draft and resume steps. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, AWS SDK for JavaScript against the Paperclip dev AWS account, a Cloudflare quick tunnel, an exploration subagent). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
e2dda16b52 |
docs(plans): record the AWS connector spike findings (#22191)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The planned AWS connector gives each connection its own AWS identity with no stored keys, using the workload identity issuer from #22083 > - The plan had open questions that only a real AWS account could answer > - This pull request records the answers from a spike against the Paperclip dev AWS account > - The benefit is that the connector build starts from proven facts, and two design details are corrected ## Linked Issues or Issue Description Refs #22083. **Subsystem affected** Connections (planned AWS connector), docs only. **Problem or motivation** The AWS plan assumed an `aws___call_aws` tool and a read-only session policy with only `ReadOnlyAccess`. Neither had been tested. **Proposed solution** Record the spike results and correct the plan. The hosted server in the test account offers `aws___run_script` but no `aws___call_aws`. A read-only session also needs `AWSMCPSignInOAuthAccessPolicy`, or the token exchange fails. **Alternatives considered** Leave the plan unchanged and find the differences during the build. Rejected: the tool choice changes the governance design. ## What Changed - `doc/plans/2026-11-17-aws-connector-and-minted-credentials.md`: a new "Spike findings" section with results and timings, and corrections in the minting, agent-call and governance sections. Phase 0 is marked done. ## Verification - Spike against the dev AWS account. The issuer code from #22083 ran locally behind a temporary public HTTPS tunnel. The spike verified these items: - AWS accepted a Paperclip-signed token. - A token for another connection was denied. - The bearer exchange worked. - `tools/list` worked. - A read and a write through `run_script` worked. - A read-only session could read, and AWS refused its write. - Both access-key paths worked. - After the spike, a listing of roles, OIDC providers, users and buckets with the spike prefix returned nothing. - Docs only. No code changed. ## Risks - Low risk. Docs only. - The tool list was observed in one account on one day. AWS can change the hosted server. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, AWS SDK for JavaScript, AWS CLI, a Cloudflare quick tunnel). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
0acac00a49 |
docs(connections): record the staging tester's move to the separate Google app (#22187)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The built-in Google Workspace connector is going through Google approval > - `doc/memory/google-connector-memory.md` holds the decisions and environment state for that work > - The staging tester now uses the separate Google app, but the doc still says it does not > - This pull request records the switch, what was verified after it, and what is still to confirm > - The benefit is that the next session starts from the true state and does not break staging deploys ## Linked Issues or Issue Description Docs are out of date after the tester moved to the separate Google app. Refs #22179. ## What Changed - `doc/memory/google-connector-memory.md`: the tester is on the separate Google app; what was verified; the consent-screen check that remains; the deploy-order warning. - `doc/plans/2026-11-17-google-workspace-production-checklist.md`: ticked "wired to the new Google app". ## Verification - Docs only. No code or tests changed. - Checked that the diff has no client ids, project ids, stack ids, passwords, or keys. ## Risks - Low. Docs only. The doc is temporary and is deleted when the connector is released. ## Model Used - Anthropic Claude Fable 5.2 (`claude-fable-5-2`), in Claude Code with tool use and browser control. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
b26c0edf23 |
docs(connections): record the staging Google tester state and findings (#22179)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The built-in Google Workspace connector is going through Google approval > - `doc/memory/google-connector-memory.md` holds the decisions and environment state for that work > - The staging tester now exists and passed its first live tests, but the doc still says "to be created" > - This pull request records the tester state, the remaining wiring order, and the test findings > - The benefit is that the next session starts from the true state and does not break staging deploys ## Linked Issues or Issue Description Docs are out of date. The working memory and the production checklist say the staging tester does not exist. Refs #22050, #22065. ## What Changed - `doc/memory/google-connector-memory.md`: tester state, what passed, what is left, and four findings from live tests. - `doc/plans/2026-11-17-google-workspace-production-checklist.md`: split the tester item into "created" (done) and "wired to the new Google app" (open). ## Verification - Docs only. No code or tests changed. - Checked that the diff has no client ids, project ids, stack ids, passwords, or keys. ## Risks - Low. Docs only. The doc is temporary and is deleted when the connector is released. ## Model Used - Anthropic Claude Fable 5.2 (`claude-fable-5-2`), in Claude Code with tool use and browser control. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
f1abee7060 |
docs(plans): record staging results and plan the production rollout (#22181)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Paperclip Cloud now sends analytics directly to PostHog, and this is live on staging > - The plan doc records the rollout state and the steps that remain > - Production needs an ordered, reversible rollout with clear decisions > - This pull request records the staging results and adds the production rollout plan > - The benefit is one reviewed place that says what is done, what is next, and how to roll back ## Linked Issues or Issue Description **What is the problem?** The plan doc still shows the staging rollout as pending and has no production steps. **What do you propose?** Update the delivery state with the merged pull requests and staging results, and add Phase 5: the production rollout, with prerequisites, one step per switch, checks, and rollbacks. **Why does it matter?** Production changes must be ordered and reversible, and the open decisions must be visible. ## What Changed - Update the delivery state table in `doc/plans/2026-11-17-cloud-direct-posthog.md`. - Add "Phase 5: production rollout". ## Verification - Documentation only. ## Risks - Low risk. Documentation only. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
11a0ff0205 |
feat(connections): per-company workload identity issuer with public signing keys (#22083)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents use outside services through connections. The control plane holds each connection's credentials and makes the calls. > - Some clouds, such as AWS, do not need stored keys. They trust a signed token from an identity provider that they can verify against its public keys. > - Paperclip has no signing key whose public half it can publish. Agent identity keys are Ed25519, which AWS does not accept, and the agent's run receives the private key. Agent run tokens use a shared secret. > - This pull request gives each company its own signing key and publishes the public half as a standard OIDC issuer. > - The benefit is that a connection, such as the planned AWS connector, can get its own short-lived cloud credentials with no stored cloud keys. The connection's role in the customer's account sets its permissions. ## Linked Issues or Issue Description Refs #19414 and #10411. Those pull requests add OIDC *sign-in* to Paperclip. This pull request is the other direction: Paperclip *issues* tokens that outside clouds verify. **Subsystem affected** Connections (tool access) and the database. **Problem or motivation** An AWS connector should not store a long-lived AWS access key. AWS supports OIDC federation (`AssumeRoleWithWebIdentity`). AWS fetches the issuer's public keys over HTTPS and checks each token. Paperclip has no key that it can publish for this, so this method is not available. **Proposed solution** Each company gets an ES256 key that Paperclip creates on first use. The vault master key encrypts the private half. The issuer is `https://{instance}/oidc/companies/{companyId}`. Paperclip publishes a discovery document and a JWKS there. Tokens last 5 minutes and have `sub = connection:{id}`. Only the control plane signs them. The board can see key state and run planned rotation or emergency rotation. The design is in `doc/plans/2026-11-17-workload-identity-issuer.md`. The AWS connector plan that uses it is in `doc/plans/2026-11-17-aws-connector-and-minted-credentials.md`. **Alternatives considered** - Reuse agent identity keys. Rejected: AWS does not accept Ed25519, and the agent's run receives the private key. - Derive the key from the master key, with no table. Rejected: the master key alone would then sign as production. Worktree environments copy the master key on purpose. A master key rotation would also change the signing key and break every customer trust. - Use one issuer for the instance. Rejected: if a customer's role checks the issuer but not `sub`, any company on a shared instance could assume that role. ## What Changed - `packages/db`: new table `company_workload_identity_keys` and migration `0498`. The migration only creates a new table, so the previous release keeps working. Partial unique indexes allow one `active` key and one `next` key per company. - `server/src/services/workload-identity.ts`: lazy key creation with concurrent convergence, a private-key cache with a fail-closed key match check, the discovery document, the JWKS, `signConnectionToken`, and rotation steps. A key that is no longer active stays published for 24 hours. Paperclip calculates this when it reads the keys, so no background job is necessary. - `server/src/routes/workload-identity.ts`: public `GET /oidc/companies/{companyId}/.well-known/openid-configuration` and `GET /oidc/companies/{companyId}/jwks.json` (404 until a key exists). Board `GET /api/companies/{companyId}/workload-identity` and `POST .../workload-identity/rotation`, with an activity log entry for each rotation. - Token issuance writes a tool-access audit event. The event never contains the token. - `cli`: worktree seeds exclude the new table in the minimal mode and the full mode. This is the same as for `agent_identity_keys`, so a development copy never holds production's signing key. - Docs: `docs/api/workload-identity.md`, an entry in the API overview, and the two plan docs. - No connector uses `signConnectionToken` yet. The AWS connector pull request will use it. ## Verification - `cd server && npx vitest run src/__tests__/workload-identity.test.ts`: 8 tests pass. The tests cover these items: - The issuer requires https with no port and no loopback host. - The endpoints return 404 before a key exists. - Six concurrent first uses create one key. - The JWKS contains only `kty crv x y kid alg use`. - A token verifies against the published key and has the expected claims. - A company cannot sign for another company's connection, and its keys do not verify that company's tokens. - Signing stops when the public address is not usable or has changed. - Two-step rotation and emergency rotation work. - Board-only access is enforced, and each rotation writes an activity log entry. - Red/green: I put back a JWK that leaks `d` and removed the company check on the connection. Three tests failed. I restored the code, and all 8 tests passed. - `cd cli && npx vitest run src/__tests__/worktree.test.ts`: 63 tests pass. - `npx tsc --noEmit` passes for `packages/db`, `server` and `cli`. - I started this branch on a real dev server. Before a key existed, the public routes returned JSON 404 (not the UI page). After an emergency rotation from the board route, the JWKS returned one public P-256 key. On the http dev instance, discovery stayed 404 and the board route reported `not_https`. - `pnpm verify` exited 2 because of the migration. The required CI gates supply the broad coverage. - Not run: a real AWS `AssumeRoleWithWebIdentity` call. That needs a public HTTPS instance. It is the first step of the AWS connector work and will run against staging. ## Risks - The migration is low risk. It creates a new table with no backfill, and the compatibility checker classifies it as compatible with the running release. - The public routes are unauthenticated. They show only public keys, return 404 for companies without a key, and use random company UUIDs. Each request is one small indexed read. - The issuer URL shape is permanent after customers trust it. A change to the instance's public address breaks those trusts until the customer updates the identity provider. `signConnectionToken` stops on that mismatch when the caller supplies the recorded issuer. - Anyone who holds the production database and the master key could sign for that instance's companies. That is the same exposure as every other vault secret. - Edge routing on Paperclip Cloud must pass `/oidc/...` to the stack. I will confirm this on staging with the AWS connector work. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, web fetch of AWS documentation, a local dev server). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
43b2d8be22 |
feat(dashboard): add an experimental areas and responsibilities dashboard (#22069)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The dashboard is the first page a person sees in a company > - The classic dashboard shows agents, runs, costs and recent tasks > - It does not show what the person is responsible for, or if that work goes well > - People think in areas, such as Fundraising or Customers, and in the things they own in each area > - This pull request adds a dashboard built on areas and responsibilities, behind an experimental setting that is off by default > - The benefit is that a person sees what they own, how it is going, and what agents do for it, on one page ## Linked Issues or Issue Description No public issue exists for this work. The description follows the feature request template. **Subsystem affected** Dashboard, navigation, the task conversation, routines, and the instance experimental settings. **Problem or motivation** The classic dashboard counts agents and tasks. It does not say what a person owns or if it goes well. People who run a company with agents want to see their areas, the responsibilities in each area, the bar each one must meet, and the agents and routines that serve it. **Proposed solution** A person owns responsibilities. Each responsibility belongs to an area and has a bar, a number to reach. Projects, tasks and routines link to a responsibility, and the page shows its status from those links. An agent can interview the person and draft the dashboard. The person accepts or declines each change, and any change can be undone from a saved version. **Alternatives considered** Add the same data to the classic dashboard. That page is a fixed set of cards for company activity, so a second, person-centred page is clearer. The setting lets a person pick one of the two. **Roadmap alignment** This is an experimental feature behind `enableResponsibilities`. It is off by default and changes nothing for a person who leaves it off. ## What Changed - Add the setting "Areas & Responsibilities dashboard" (`enableResponsibilities`, off by default). On, Dashboard in the sidebar opens the new page, and the classic dashboard link is not shown in the sidebars or the command palette. Off, nothing changes. - Add tables and migrations 0492 to 0497 for areas, responsibilities, links to work, metrics, status reports, saved dashboard versions, and model-call records for summaries. - Add the API: areas, responsibilities, links, bars, metrics, status reports, summaries, runs, automation, and the dashboard conversation with versions and undo. Each route checks company access and writes to the activity log. - Add four task conversations with an agent: about a bar, about an automation, about a routine, and about the whole dashboard. The agent proposes a change. Nothing applies until a person accepts it. - Add standing checks: routines that an agent carries out on its heartbeat and reports. - Add the Dashboard page: search, counts, areas that open and close, a primary New area button, and a message box at the bottom of the page. Add the responsibility page with its bar, metrics, work and routines. - Add the Paperclip skill sections and runner API entries that let an agent use these routes. The runner skill inventory is regenerated (166 to 174 rows). - Keep the new runner API tests in their own file so `runner-api.test.ts` stays at 985 lines. - Onboarding is not changed. The onboarding changes that build on this work are a separate pull request. ## Verification - Type checks pass for `packages/db`, `packages/shared`, `server` (all layers) and `ui`. - `pnpm --filter @paperclipai/db check:migrations` passes. A new `drizzle-kit generate` run produces no further change. - Tests for every changed file pass: 26 server test files (3,009 tests), 116 UI test files (1,991 tests) and 28 shared test files (432 tests). - The runner capability inventory checks pass. - I started the app on a local database that already held earlier data. Master's new migrations and these migrations applied without error. With the setting on, the Dashboard and a responsibility page loaded with no console or server errors. The More menu and the command palette show one Dashboard. - I did not run `pnpm verify`, the full local suite, or the browser suites. Required CI covers them. - The agent conversations are tested with simulated agents. No real model has run them. ## Risks - Medium. The change is large, and the setting is off by default. With it off, the new routes refuse requests and the UI is unchanged. - Migrations only add tables and columns. Two replace a CHECK constraint, and one builds an index `CONCURRENTLY`. These two have the `online_compatible` override with a reason in the file, and the second one is listed in `MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS`. - Master can add migrations before this merges. The migrations then need new numbers. - `responsibilities.ts` (1,663 lines), `responsibility-summaries.ts` (1,239 lines) and one Storybook fixture file (1,305 lines) are new files over 1,000 lines. They each keep one cohesive unit together: the service for one table family, the model-call path for summaries, and the fixtures that the stories share. No existing file crosses 1,000 lines. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Anthropic Claude: Claude Fable 5.2 (`claude-fable-5-2`) wrote most of the change, with Claude Sonnet 5.5 (`claude-sonnet-5-5`) for the final steps. Both used tool use, code execution and browser checks. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Scott Tong <scott@scottsmbpm5max.lan> Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
56023fe80f |
feat(google-workspace): gateway sets the REST tool result size limit for all six services; update connector docs (#22050)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents read Gmail, Calendar, Drive, Docs, Sheets and Slides through the built-in Google Workspace connection > - Paperclip now has its own REST-backed Google tools in `server/src/google-workspace-rest-tools/`. No service uses them yet > - That folder guessed a result size limit of 250,000 bytes. It applied the limit only to Drive, Docs, Sheets and Slides. Gmail and Calendar had no limit > - The guess was wrong in both directions. It refused documents that Paperclip can deliver, and the folder should not know Paperclip's limits > - This pull request makes the limit a number that the gateway gives. The folder applies it to every tool in all six services through one check. It also brings three Google planning docs up to date > - The benefit is that a REST result is refused only when the gateway would also refuse the same result from Google's own MCP server ## Linked Issues or Issue Description Refs #22027, #22028, #22029, #22026 **Is your feature request related to a problem?** The REST tools refused results over a size the folder chose itself (250,000 bytes). Paperclip can deliver larger results: the runner saves a large result as an artifact and gives the agent a pointer. Gmail and Calendar results had no limit at all. **Describe the solution you'd like** The gateway gives the limit. The folder applies the given number to the final serialized result of every tool, with one helper. A result over the limit is an error that says how to ask for less. It is never cut short. **Describe alternatives you've considered** A larger number chosen from the runner's 1 GiB capture limit. Google's own transfer limit (20 MB per response) and memory use make that a poor choice. Also, results would then differ between the REST path and Google's MCP path. ### Limits a tool result passes on the way to an agent | Point | Limit | What happens over it | | --- | --- | --- | | Google response read by `google-http.ts` | 20 MB (`MAX_RESPONSE_BYTES`) | Tool error. Nothing returned | | REST folder result (new) | the number the gateway passes: 999,885 bytes | Tool error with "read less" advice. The gateway returns `tool_error` (502) | | Gateway read of a remote MCP response (`MAX_REMOTE_MCP_RESPONSE_BYTES` in `tool-gateway.ts`) | 1,000,000 bytes per JSON body or event-stream message | `mcp_remote_response_too_large` (502). This is the "1,000,000 bytes are rejected" statement in `doc/prompt-injection-prevention.md`. It applies to the Google MCP path only; the REST path has no HTTP read | | Injection screening model check | 256 KiB model request | Not a delivery limit. A larger result is checked by the built-in patterns only | | Gateway HTTP route (`POST /tool-gateway/.../tools/call`) | none in Paperclip | `res.json(result)` | | Runner (`captureRunnerToolResult`) | 500,000 bytes (`RUNNER_TOOL_RESULT_MAX_BYTES`) | Saved as an artifact. The agent gets a pointer and reads it with `read_tool_result` | | Runner capture | 1 GiB (`RUNNER_API_RESPONSE_MAX_BYTES`) | `SemanticToolOutcomeUnknownError` | The gateway wraps the result as `{ content, data: { content, structuredContent, ... } }`, so the payload occurs again. Measured: a Gmail result of 676,459 bytes from the folder became 1,017,386 bytes delivered (ratio 1.50). The ratio cannot be more than 2. ### The budget `GOOGLE_WORKSPACE_REST_MAX_RESULT_BYTES` in `tool-gateway.ts` = `MAX_REMOTE_MCP_RESPONSE_BYTES` less the JSON-RPC envelope and event-stream framing (`event: message\ndata: {"jsonrpc":"2.0","id":"paperclip-tool-<uuid>-version","result":...}`) = **999,885 bytes**. A result that Google's MCP server could deliver through the gateway is also delivered over REST, and the reverse. With the wrapper, the largest delivered result is less than 2 MB. The runner saves it as an artifact (far below 1 GiB). So the folder never refuses a result that the runner can give as a pointer. Practical limits at that budget (plain ASCII; escaped and non-ASCII characters count more): | Tool | Largest content | | --- | --- | | Gmail `get_message` | about 500 KB plain body, or about 250 KB each for plain plus HTML | | Calendar `get_event` | about 500 KB description | | Drive `read_file_content` | about 500 KB text | | Drive `download_file_content` | about 375 KB file bytes | | Docs `read_doc` | about 240 KB document text | | Slides `read_presentation` | about 250 KB text | | Sheets `get_values` | about 500 KB cell text, at most 50,000 cells | ## What Changed - `tool.ts`: `defineTool` takes `whenTooLarge` advice per read tool (required for reads, refused for writes) and a `ToolLimits` argument. It measures the final `CallToolResult` once (`withinResultBudget`) and replaces an oversized result with a `too_large` tool error. `FALLBACK_MAX_RESULT_BYTES` (250,000) applies only when a direct caller gives no limit. - `callGoogleWorkspaceRestTool` and `createGoogleWorkspaceRestServer` take `maxResultBytes`. A value that is not a positive integer is refused before any request. - All 23 read tools state their advice: a smaller `pageSize` for list tools, `messageFormat` or `view` for Gmail, a narrower range for Sheets, `fields` for Slides. Write tools are not checked: a completed write is never reported as an error, so an agent is not led to repeat it. - Drive, Docs, Slides and Sheets no longer have their own byte checks (`boundedStructuredResult`, `boundedResult`, `sizedResult`, `MAX_CONTENT_BYTES` removed). Drive downloads stop reading at the given limit. The Sheets 50,000-cell limit stays. - Two Drive tool descriptions no longer state a fixed number. - Adapter `google-workspace-rest-dispatch.ts`: `maxResultBytes` is a required input and is passed through. - `tool-gateway.ts`: new `GOOGLE_WORKSPACE_REST_MAX_RESULT_BYTES`, passed on the REST branch. - Folder README: new "Result size" section with the limits and costs; old 250,000-byte text replaced. - Docs: `doc/memory/google-connector-memory.md`, the production checklist and the code-fix plan now list #22026 to #22029, the REST status (36 of 36, none switched on), the November 17 decisions, the known differences from Google's tools, open risks and next steps. ## Verification - `cd server && npx vitest run src/google-workspace-rest-tools` passes (53 files, 1,361 tests with the adapter test). - New `result-budget.test.ts`: for Gmail (read, list), Calendar (list, read), Drive, Docs, Slides and Sheets, a result just under a given budget is returned and one just over is refused with the tool's advice. Without a budget, the fallback applies; with a larger budget, the same result is returned. Drive downloads use the given cap. A Gmail list tells the caller to lower `pageSize`. A Gmail draft result over the budget is still returned, not an error. `defineTool` refuses a read tool without advice and a write tool with it. - `npx vitest run src/services/google-workspace-rest-dispatch.test.ts`: the adapter passes the gateway's number, and the folder applies exactly that number. - `npx vitest run src/__tests__/google-workspace-rest-transport.test.ts` (24 tests): with Gmail on REST, a 330 KB email passes the gateway with screening, is more than 500,000 bytes delivered, and `captureRunnerToolResult` turns it into a `read_tool_result` pointer. A 520 KB email is refused with `tool_error` and the advice, after one Google request. - `npx tsc --noEmit -p server/tsconfig.json`: no errors. - `pnpm verify --explain`: broader verification required (a doc under `doc/` is a global CI input). CI supplies that coverage. - Time per call: the check is one `JSON.stringify` of the result. About 5 µs for a 4 KB result, 18 µs for 42 KB, 0.24 ms for 700 KB. Drive, Docs, Sheets and Slides had the same pass before, so they add nothing. Gmail and Calendar add this one pass. ## Risks - Gmail and Calendar results now have a limit. Direct callers that pass no limit get the 250,000-byte fallback. The gateway always passes its number. - Drive, Docs, Slides and Sheets accept results up to about 4 times larger than before through the gateway. Large results reach agents as artifact pointers. - Error text for oversized results changed to one shared form. - No service is switched to REST, so users see no change today. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code with tool use (shell, file edits, GitHub CLI) and extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
7f1f0eef67 |
fix(google-workspace): request free/busy with read & write Calendar, and update connector docs (#22026)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The built-in Google Workspace connector gives agents Gmail, Drive and Calendar through one connection, behind an experimental setting > - The user picks each service as read only or read & write. Paperclip then requests only scopes from a fixed list of 14 > - Read & write Calendar requested `calendar.calendarlist.readonly` and `calendar.events`. Our code assumed `calendar.events` also covers free/busy lookups > - Google's REST `freeBusy.query` does not accept `calendar.events`. So the REST `suggest_time` tool cannot read free/busy on a read & write grant, and falls back to reading each attendee's events > - This pull request adds `calendar.events.freebusy` (already one of the 14) to the read & write Calendar selection of the unified connection, and stops treating `calendar.events` as implying free/busy > - The benefit is that `suggest_time` on REST can see coworkers who share only free/busy, and the stored grant no longer claims a permission that Google did not give. It also brings three Google connector planning docs up to date ## Linked Issues or Issue Description Refs #22016, #22017, #22010, #22014 **What happened?** The read & write Calendar selection of the unified Google Workspace connection requests `calendar.calendarlist.readonly` and `calendar.events`. `toolScopes.calendar.suggest_time` lists `calendar.events` as sufficient, and `googleWorkspaceEffectiveScopes` treats `calendar.events` as implying `calendar.events.freebusy`. Google's two references differ: - REST [`freeBusy.query`](https://developers.google.com/workspace/calendar/api/v3/reference/freebusy/query), checked 2026-11-17: "This request allows authorization with at least one of the following scopes:" `calendar.readonly`, `calendar`, `calendar.events.freebusy`, `calendar.freebusy`. `calendar.events` is not in the list. - Google's hosted Calendar MCP [`suggest_time`](https://developers.google.com/workspace/calendar/api/v3/reference/mcp/tools_list/suggest_time), checked 2026-11-17: "Requires one of the following OAuth scopes:" `calendar`, `calendar.readonly`, `calendar.events`, `calendar.events.freebusy`, `calendar.freebusy`. Here `calendar.events` is accepted. So `calendar.events` is enough for Google's MCP tool but not for the REST endpoint that our REST `suggest_time` uses. **Expected behavior** A user who picks read & write Calendar gets a grant that allows `freeBusy.query`. A grant that does not have the free/busy permission is not recorded as having it. Grants made before this change keep working as read & write. **Steps to reproduce** 1. Call `googleWorkspaceSelectedScopes({ calendar: "write" })`. It returns two scopes, without `calendar.events.freebusy`. 2. Call `googleWorkspaceEffectiveScopes(readOnlyCalendarScopes, [calendar.events])`. It returns `calendar.events.freebusy`, which Google did not grant. 3. With Calendar on the REST transport, `suggest_time` on a read & write grant gets an insufficient-scope answer from `freeBusy.query` and uses the events fallback. ## What Changed - `packages/shared/src/google-workspace-connectors.ts`: - `googleWorkspaceSelectedScopes` adds selection-level scopes from a new `SELECTION_ADDED_SCOPES` map. Read & write Calendar adds `calendar.events.freebusy`. The per-service `calendar.write` profile (`SERVICE_PROFILES["calendar.write"]`) is unchanged, because the Cloud broker matches legacy profiles by exact scope set. - `googleWorkspaceEffectiveScopes` no longer expands `calendar.events` to `calendar.events.freebusy`. `calendar.events` still implies `calendar.events.readonly`. `calendar.readonly` still implies free/busy, which Google's reference allows. - `googleWorkspaceSelectionForScopes` is unchanged. It needs the legacy write profile's scopes, so a two-scope grant and a three-scope grant both read back as read & write. - `toolScopes.calendar.suggest_time` is unchanged. It still accepts `calendar.events`: Google's MCP tool accepts it, and on REST the tool has an events-based fallback that names attendees it cannot read. - Counts: default all-read-and-write selection is now 10 scopes (was 9); all read-only stays 8; the union of all 27 selections stays exactly the 14. - Tests (written first and seen failing on `master`): - `packages/shared/src/google-workspace-connectors.test.ts`: scopes for read and for read & write Calendar; the legacy `calendar.write` profile keeps two scopes; default count 10 and read-only count 8; old two-scope grant, new three-scope grant and partial consent (user declines free/busy) all round-trip as read & write and keep `list_calendars`, `list_events`, `suggest_time`, `create_event` and `delete_event`; no recorded free/busy that Google did not grant. The existing exhaustive test still checks that no selection requests a scope outside the 14. - `server/src/__tests__/tool-access-service.test.ts`: customer-owned Workspace OAuth with read & write Calendar asks Google for all three scopes. With and without free/busy in Google's answer, the connection is active, the Calendar tools are listed, the grant stores exactly what Google granted, and it reads back as read & write. - Stale tests on `master` fixed: `#22014` made Gmail 9 of 9 in the REST tool set while `#22016` still expected 3 Gmail REST tools. `google-workspace-rest-dispatch.test.ts` and `google-workspace-rest-transport.test.ts` now derive the Gmail list from the allowlist and use Chat as the service with no REST tools. - Docs: `doc/connections/GOOGLE-WORKSPACE.md`, `doc/connections/GOOGLE-WORKSPACE-SERVICE-CONSENT.md` and `doc/connections/tool-method-permission-reviews.json` say ten scopes and explain the Calendar change. - Docs: `doc/memory/google-connector-memory.md` (shipped PRs #22010, #22014, #22016, #22017; three Nov 17 decisions; "Where Google content is kept" replaced with #22010's inventory, what deletion keeps and its limits; Scopes section; REST tools status table; still-open list), `doc/plans/2026-11-17-google-workspace-production-checklist.md` (help page and retention audit ticked with #22010; REST tool progress), `doc/plans/2026-11-17-google-workspace-code-fixes.md` (fix 6 done with limits; section 9 status table; fix 10 note). - `tests/runner-e2e/source-sdk-public-inputs.json`: refreshed pins for the changed shared files. Not changed: - `server/src/google-workspace-rest-tools/**`. Another change owns that folder. Its README open question about this scope is now resolved and can be removed there. - The picker disclosure (`googleWorkspaceDisclosure.ts`). Read & write Calendar already says agents "can see your calendars, events and free/busy times". With this change that line is accurate on REST too. Behavior for existing grants: a saved read & write grant without free/busy still reads back as read & write, keeps every Calendar tool, and refreshes without change. Nothing prompts the user to reconnect. A user who chooses read & write Calendar in the picker is asked for the new scope. An account-page reconnect without a new choice still restores the exact saved request, as before. Cloud broker and Google Console: no change needed. `calendar.events.freebusy` is one of the 14 in `workspace.all`, the broker accepts any nonempty subset of `workspace.all` (`doc/connections/GOOGLE-WORKSPACE-SERVICE-CONSENT.md`), and the legacy `calendar.write` profile's exact scope set did not change. ## Verification - `cd packages/shared && npx vitest run src/google-workspace-connectors.test.ts src/app-definitions.test.ts`: 55 passed. On `master` the new shared tests fail (3 failed). - `cd server && npx vitest run src/__tests__/tool-access-service.test.ts -t "free/busy"`: 2 passed. With `master`'s `google-workspace-connectors.ts`: 2 failed. - `cd server && npx vitest run src/__tests__/google-workspace-rest-transport.test.ts src/services/google-workspace-rest-dispatch.test.ts`: 29 passed (4 failed on `master` before this change). - `cd ui && npx vitest run src/features/connections/GoogleWorkspaceServices.test.tsx src/pages/apps/AppDetail.test.tsx`: 115 passed. - `cd packages/shared && npx tsc --noEmit -p .`: clean. - `pnpm verify` exits 2 (a global CI input changed), so required CI supplies the broad coverage. - Docs facts checked against `gh pr view` for #22010, #22014, #22016, #22017, the code, and paperclipai/paperclip-website PR 197 (still a draft; does not mention OpenRouter). ## Risks - Low risk. The submitted set of 14 does not change, and the broker contract does not change. - New read & write Calendar grants record `calendar.events.freebusy` next to `calendar.events`. Code that reads stored scopes sees one more scope. All consumers (callback, refresh, discovery, picker pre-fill) are covered by the tests above. - A read-only Calendar request on an account that Google reports only `calendar.events` for no longer records free/busy. That matches what Google granted; `suggest_time` is then not offered on that grant. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use, web fetch and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
1ca90d2d6f |
docs(connections): record Nov 17 Google Workspace progress in memory, checklist and plan (#22007)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The built-in Google Workspace connector is going through Google's restricted-scope review > - Three docs track that work: the working memory, the production checklist and the code-fix plan > - Four pull requests merged on November 17 and one more is open, so the docs are out of date > - This pull request updates the three docs to match the merged work and the Cloud audit results > - The benefit is one current place for decisions, scopes, shipped work and open questions ## Linked Issues or Issue Description Refs #21996, #21998, #21999, #22000, #22001 **Issue type** Outdated documentation. **Where is the issue?** `doc/memory/google-connector-memory.md`, `doc/plans/2026-11-17-google-workspace-production-checklist.md` and `doc/plans/2026-11-17-google-workspace-code-fixes.md`. **What's wrong?** The docs describe fixes 1 to 4 as not done. They plan a new npm package for the REST tool set, but the tool set is a folder inside the server. The scope table is in two files. The docs do not record the Cloud broker audit or where Google content is kept. **Suggested fix** Record what shipped, add the new decisions, keep one scope table, add the broker and data-retention findings (marked as reported), and tick the checklist items that are now true. ## What Changed - Working memory: three new decisions (no new npm packages, the scope handling contract, agents merge their PRs on the coordinator's approval at green checks and Greptile 5/5). - Working memory: new "What has shipped" table (#21996, #21998, #21999, #22001 and #22000, with merged commits). - Working memory: the single copy of the 14-scope table. "Known mismatches" now lists only what is still true after #22001. - Working memory: new "Cloud broker (paperclip-cloud)" and "Where Google content is kept" sections, from the November 17 read-only audit, marked as reported and not re-verified. - Working memory: how to create the staging tester, its two obstacles, and two new open decisions with a recommendation. - Working memory: the privacy policy draft location and what it must also cover (the OpenRouter screening model and the picker's retention statement). - Working memory: a note that the ponytail reviewer flagged the summary of Google's reply. The summary stays. - Checklist: the scope table is now one sentence with a link. Four product items are ticked with date and PR link. Items that are not fully true stay unticked. - Code-fix plan: fixes 1 to 4 marked done with PR links. Section 9 layout is now the folder inside the server. Two one-line lessons. Section 10 lists the broker changes and the deploy order. ## Verification - Each fact was checked against `gh pr view` for #21996, #21998, #21999, #22000 and #22001, and against `packages/shared/src/google-workspace-connectors.ts` and `server/src/services/tool-access/oauth/grant-refresh.ts` on `origin/master`. - The broker and data-retention facts come from an audit that this PR does not re-verify. The docs say so. - Searched the changed files for client IDs, secrets and project numbers: none. - Docs only. No tests apply. ## Risks - Low risk. Docs only. No code or behavior changes. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits, GitHub CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
3293700448 |
docs(plans): Cloud sends analytics directly to PostHog (#22003)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Paperclip Cloud is the hosted version, and it shares the product telemetry path with self-hosted installs > - Self-hosted telemetry goes through the Paperclip collector, which anonymizes every identifier > - Cloud needs real identifiers, page analytics, and session replay in PostHog to find and fix bugs for specific users > - This pull request adds the plan that moves Cloud to direct PostHog delivery and keeps the self-hosted path unchanged > - The benefit is one reviewed plan that the follow-up pull requests implement ## Linked Issues or Issue Description **What is the problem?** The PostHog Product analytics and Web analytics pages show no data for Paperclip Cloud. Only server events arrive, and they carry anonymized install identifiers. **What do you propose?** Cloud sends events directly to PostHog with real identifiers. Self-hosted installs keep the collector path. Existing Cloud stacks keep their PostHog history through a precomputed legacy identifier. **Why does it matter?** The team can trace a bug from a specific user or workspace to events and session replays. ## What Changed - Add `doc/plans/2026-11-17-cloud-direct-posthog.md`. ## Verification - Documentation only. No code changes. ## Risks - Low risk. Documentation only. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
12cb18e17d |
docs(connections): Google Workspace approval checklist, code-fix plan and working memory (#21996)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The built-in Google Workspace connector lets agents read and write Gmail, Drive, Docs, Sheets, Slides and Calendar data > - The connector stays behind an experimental setting until Google approves its data access and the security assessment is accepted > - The approval work has many steps, and the decisions behind them were spread across chats and private notes > - This pull request adds one checklist, one code-fix plan and one temporary working-memory file for that effort > - The benefit is that contributors and agents share one source for the scope contract, the decisions and the remaining work ## Linked Issues or Issue Description **Type of issue** Missing documentation for an in-progress effort. **Description** The Google Workspace connector needs Google approval before it can go to all users. There is no single document that lists what "approved and released" means, which code changes must land before submission, or the decisions taken so far (launch transport, owner, test instance, 14-scope contract). **Proposed change** Add a production and approval checklist, a code-fix plan for section 2 of that checklist, and a temporary working-memory file. Keep identifiers, credentials and review correspondence out of the public repository. ## What Changed - Added `doc/plans/2026-11-17-google-workspace-production-checklist.md`: the "done means" criteria, the 14-scope contract, and checklists for decisions, baseline, product and code, policy, security assessment, reviewer environment, live acceptance, demo video, submission, release and ongoing work. - Added `doc/plans/2026-11-17-google-workspace-code-fixes.md`: ten ordered code fixes (disclosure in the picker, scope lock to 14, `gmail.modify` removal from the managed client, injection screening check, revoke on last disconnect, retention audit, scan findings, gate removal, REST-backed tool set, scope cleanup). Each item is marked confirmed or to check. - Added `doc/memory/google-connector-memory.md`: a temporary file with decisions, open questions, environments, Google app layout, scopes, review history and privacy-policy gaps. It says to delete it once the connector is released. ## Verification - Docs only. No code, schema or test changes. - Read all three files before the push and confirmed that they contain no passwords, client IDs, client secrets, project numbers or review login codes. - Relative links between the three files and to `doc/connections/GOOGLE-WORKSPACE.md` resolve in the repository. - CI checks on this PR. ## Risks - Low risk. Documentation only. - The plans describe intended work. Some items are marked "to check" and can change after the code is traced. - The working-memory file is temporary and must be deleted after release. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), used through Claude Code with tool use (shell, git, GitHub CLI) to commit the files and open this PR. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
3706f520fd |
test(scaling): keep ramping through recovered runs, read spend from cost records, and wait for termination before deleting agents (#21776)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The scaling effort uses a real-agent ramp harness (`scripts/scale-agents-ramp.ts`, #21695 and #21757) to load a hosted test instance through the product API > - Its first run with agents that really ran (10 agents, 105 tasks, 106 runs) stopped on one failed run, recorded $0 model spend, and left cleanup errors > - Read-only inspection of the instance showed three harness faults. The failed run had a same-run bootstrap retry scheduled 33 s later, but the harness stopped and cleaned up first. Run rows carry no cost, but cost records do (63.7 cents for the run's agents). Terminated agents leave the company agent list, so the harness treated them as done and sent deletes that Paperclip refused with 409 > - These faults make the ramp stop too early, make its spend cap inert, and make cleanup report the wrong cause > - This pull request keeps the ramp going through runs that Paperclip recovers by itself, reads spend from cost records, records board latency per route, and makes cleanup wait for termination and name what it left behind > - The benefit is that the next ramp grades the product, not the harness, and its spend cap works ## Linked Issues or Issue Description Refs #21695 Refs #21757 **What happened?** The ramp stopped at step 1 with `failed_runs_zero failed (observed 1)`. The run failed with `runner_ingress_bind_conflict` before its provider session opened. Its run events show `same-run resume persisted` with `recoverable: true` and `nextAttemptAt` 33 s later. The harness stopped within 9 s and paused the agents, so the retry was blocked (`native_retry_admission_ineligible`). The result showed `Recorded model spend: $0`. The harness read `costUsd` from run rows, which is always null; `GET /api/companies/<id>/costs/by-agent` holds 63.71 cents over 104 cost events for the 10 agents. Cleanup reported two `DELETE /api/agents/:id returned 409: Complete termination before deleting the agent`. Those agents were still `lifecycleState: terminating`; the harness waited on the company agent list, which hides terminated agents, so it treated them as done. A later `--cleanup-only` changed nothing because it also found agents only through that list. The board API p90 failed (765 ms), but the result had no per-route timings to say which request was slow. **Expected behavior** A failed run whose task finishes by itself is counted, not a stop. A failed run whose task does not finish within a grace stops the ramp. The share of failed runs is graded at the step end. Spend comes from Paperclip's cost records. Cleanup deletes an agent only after termination completes, and names the agents and sandbox leases still pending. **Steps to reproduce** 1. Run `node --test scripts/scale-ramp/scale-ramp.test.ts` with the new fake and tests and the `origin/master` `ramp.ts`, `cleanup.ts`, `criteria.ts` and `report.ts`. 2. 16 tests fail, among them every new test (recovered run, unrecovered run, spend from cost records, spend cap, board per route, termination wait, still-terminating agents and `--cleanup-only`, leases released during the wait). 3. With this change, all 60 pass. **Paperclip version or commit** Instance on `2026.1106.0+2564.git.3244780bd`; harness from #21757. ## What Changed - `criteria.ts`: replace `failed_runs_zero` with `failed_runs_unrecovered_zero` (immediate) and `failed_run_rate` (at the step end). Step metrics add `runsEnded`, `runsFailedRecovered`, `runsFailedUnrecovered`, `runsFailedPending` and `board.byRoute`. - `ramp.ts`: a run counts as failed once it was seen `failed` or `timed_out`. It is recovered when its task reaches `done`. It is unrecovered when the task rests not done, needs Retry, or has no live run for `--recovery-grace-minutes` after the failure. A task whose interruption has an automatic retry scheduled (`state: retry_scheduled`, `nextRetryAt` or `retryWait`) does not count as needing Retry. Spend comes from `costs/by-agent` for the run's agents on each poll, and each step reports the spend in its window. Board probes keep a distribution per route. - `config.ts`: add `--max-failed-run-percent` (default 2) and `--recovery-grace-minutes` (default 15). - `cleanup.ts`: read agents by id as well as from the list, so terminated agents in the state file are found. Wait for `lifecycleState` `terminated` or `rejected` by id before a delete. Report `agentsStillTerminating`, `activeSandboxLeaseIds` and `sandboxLeasesReleasedDuringWait`, and name each lease still active with its run and acquire time. - `report.ts`: the summary shows recovered, unrecovered and pending failed runs, board latency per route, and each cleanup error. - `fake-paperclip.ts`: run rows carry no cost; `costs/by-agent` returns cents per agent. Optional failure of the n-th run, a scheduled recovery run, a termination delay (terminated agents leave the list and a delete answers 409 until termination completes), and leases that stay active after their run. - `doc/plans/2026-11-13-thousand-concurrent-agents.md`: the runbook describes the failed-run criteria, the spend source and the cleanup behavior. ## Verification - `node --test scripts/scale-ramp/scale-ramp.test.ts`: 60 of 60 pass. - Red check: with the new fake and tests but the `origin/master` `ramp.ts`, `cleanup.ts`, `criteria.ts` and `report.ts`, 16 tests fail, including all 8 new ones. - `tsc -p scripts/scale-ramp/tsconfig.json`: no errors. - Evidence from read-only GETs on the test instance: run `f1892d43` events (bind conflict, scheduled same-run retry, blocked after the pause), run rows (`costUsd` null on all 106), `costs/by-agent` (63.71 cents for the 10 agents), agent rows by id (`terminating` while absent from the list), lease rows. - Not run: a new real ramp (needs the owner's go-ahead). ## Risks - Low risk. The change touches only the scale-ramp harness, its fake, its tests and one runbook section. No product code changes. - A failed run no longer stops the ramp at once. An unrecovered one stops it after the grace (15 min by default), so a broken step can run up to that long. The step-end failed-run rate still fails a step with more than 2% failed runs. - Spend now needs cost read access for the board token. If `costs/by-agent` cannot be read, the result reports spend as not measured and the spend cap does not trigger. - Cleanup can now take up to 5 more minutes while it waits for termination by id. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, read-only HTTP GETs to the test instance). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
15afc32137 |
test(scaling): wait for agent setup and gate the ramp on a first started run (#21757)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The scaling effort uses a real-agent ramp harness (`scripts/scale-agents-ramp.ts`, added in #21695) to load a hosted test instance through the product API > - Its first real run held a 20-minute step with 10 tasks, 10 queued runs and 0 started runs, and reported only "concurrency_reached failed (observed 0)" > - The cause was in the harness: it bound the agents to a plugin Runner-task environment, which runs only agents with a managed AI connection. Each agent stayed in setup (status `paused`, setup state `verifying`) with that reason, so each run stayed queued. The product did what it is designed to do > - The harness did not wait for agent setup, did not read the setup error, and held the full step with nothing running. Its fake API created agents ready at once, so its tests could not catch this > - This pull request makes the harness wait for agent setup, gate the ramp on one started run, and stop within minutes with Paperclip's own reason > - The benefit is that a wrong environment or any other start blocker costs about one minute and prints the cause, not a 20-minute step with no explanation ## Linked Issues or Issue Description Refs #21695 **What happened?** A ramp of 10 then 50 agents on a hosted test stack created 10 agents, 10 tasks and 10 runs. In 20 minutes no run started. The result said only "concurrency_reached failed (observed 0)". Cleanup found every agent already paused and cancelled 10 queued runs. The tasks never left `todo` (status version 1, no checkout run). The agents were bound to the "Paperclip Sandbox (Daytona)" plugin environment, whose `daytona-tasks` driver declares `supportsTasks: true`. For such an environment, lifecycle verification refuses a `paperclip_runner` agent that has no `runtimeConfig.aiConnection` (`ai_connection_required`, see `server/src/services/agent-harness-verification.ts` and its test "refuses a plugin Runner-task agent without a managed AI connection before allocating a target"). The failure pauses automatic verification, so the agents never become ready. **Expected behavior** The harness must not create a full step of work before agents can run. When agents cannot finish setup, or when no run starts, it must stop soon, print the task, run and agent state with Paperclip's reason, and clean up. **Steps to reproduce** 1. Run `node --test scripts/scale-ramp/scale-ramp.test.ts` with the new fake (agents start in setup; `agentSetupError` keeps them there) and the old `ramp.ts`, `setup.ts` and `config.ts`. 2. The new test "stops within minutes with Paperclip's reason when the agents never finish setup" fails with `concurrency_reached`, the same result as the real run. 3. With this change, it stops with `agents_ready` and the setup reason in under 5 virtual minutes. **Paperclip version or commit** Instance on `2026.1106.0+2555.git.819f61f1f` (master at 11:34Z on 2026-11-15); harness from #21695. ## What Changed - `setup.ts`: add `waitForAgentsReady`, `agentReady` and `describeAgent`. The harness reads the company agent list until each agent has setup state `ready` and is not paused. It stops when an agent reports a setup error for 60 s, or after `--agent-ready-minutes` (default 10), and prints the agent's status, setup state, holds and Paperclip's reason. - `ramp.ts`: before the first step, a preflight creates one agent and one task and creates the rest only after that task's run starts. A new check stops the ramp when no run has started within `--start-timeout-minutes` (default 5) of the first task. The stop reason names the task status, any execution blocker, the run status, error code, error and liveness, and the agent's setup state. New stop criteria: `agents_ready` and `run_started`. Run records keep `error`, `errorCode` and `livenessReason`. - `config.ts`: add `--start-timeout-minutes` and `--agent-ready-minutes`. - `fake-paperclip.ts`: agents start as the real product creates them (status `paused`, setup state `preparing`) and become `ready` after `agentReadyDelayMs`, or stay in `verifying` with `agentSetupError`. Issue creation queues a run even for an agent in setup, as the real product does. Runs start only for a ready agent. - `scale-ramp.test.ts`: three tests. Agents that never finish setup stop the run with `agents_ready` and the reason, with one agent and no task created. A first task that never starts stops the run at the preflight with `run_started`, the task, run and agent state, and one cancelled run. The rest of a step is created only after the preflight task starts. - `doc/plans/2026-11-13-thousand-concurrent-agents.md`: the Phase 3 runbook describes the start checks, the first run's cause, and which environments the harness can use. ## Verification - `node --test scripts/scale-ramp/scale-ramp.test.ts`: 43 of 43 pass. - Red check: with the new fake and tests but the `origin/master` `ramp.ts`, `setup.ts` and `config.ts`, 3 tests fail. The setup test fails with `concurrency_reached` (the real run's result) where `agents_ready` is expected. - `tsc -p scripts/scale-ramp/tsconfig.json` (same options): no errors. - Read-only inspection of the hosted instance (GET only): task rows (`todo`, status version 1, no checkout run), activity (agent created, no pause, 10 `heartbeat.cancelled` at cleanup), the plugin manifest (`daytona-tasks` `supportsTasks: true`), and the company AI connections (no OpenRouter connection). The run and agent rows were deleted with the agents at cleanup, so their setup error was inferred from the code path and the cleanup counts, not read. - Not run: a new real ramp (needs the owner's go-ahead). ## Risks - Low risk. The change touches only the scale-ramp harness, its fake and its tests, and one runbook section. No product code changes. - A real ramp now spends up to one preflight start (seconds to minutes) before the first step. The preflight task is part of step 1 and the preflight running samples are dropped, so step metrics keep their meaning. - An agent whose setup error clears within 60 s does not stop the run. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use (shell, file edits, read-only HTTP GETs to the test instance). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
b5ba548112 |
feat(server): add an instance-admin server metrics endpoint for the scale ramp (#21708)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The scaling plan (`doc/plans/2026-11-13-thousand-concurrent-agents.md`) has a finish line that a real-agent ramp harness checks (#21695) > - Four finish-line criteria need numbers from inside each server process: main-thread CPU, run event lateness, agent API time before the handler, and missed lease renewals > - The server had no way to give them, so every harness run was at best "incomplete" > - This pull request adds one instance-admin endpoint that returns those numbers per process over a short sliding window, off by default, and connects the harness to it > - The benefit is that a ramp run can measure every server criterion it can see, with near-zero cost when on and no cost when off ## Linked Issues or Issue Description Refs #21695 **Subsystem affected** Server (instance routes, run event append, native lease renewal), database client (statement counter), and the scale-ramp harness in `scripts/scale-ramp/`. **Problem or motivation** The scale-ramp harness reads `PAPERCLIP_SCALE_METRICS_URL`, but the server has no endpoint for it. The criteria `main_thread_cpu`, `run_event_lateness_p99`, `agent_api_pre_handler_p99` and `lease_renewals_missed_zero` are therefore always "not measured", and no run can pass. **Proposed solution** Add `GET /api/instance/server-metrics`. It returns a JSON snapshot of one server process over the last 50 to 60 s. Only an instance admin can read it. The env flag `PAPERCLIP_SERVER_METRICS` (read once at start, default off) turns on collection and the route. Off, the path answers 404 and no collector starts. **Alternatives considered** - The postgres.js `debug` option for the statement counter: rejected, because it captures a stack (`new Error()`) for every query. - Process CPU from `process.cpuUsage()` as `mainThreadCpu`: rejected, because it includes GC and libuv threads. It is returned as `processCpu` instead, and `mainThreadCpu` reads the main thread from `/proc` (Linux only, null elsewhere). - Default on: the measured cost is negligible (numbers below), but the event-loop monitor wakes the process every 20 ms when idle, and this is a new instance-wide surface. It stays off; the runbook says how to turn it on for a ramp run. ## What Changed - `server/src/services/server-metrics.ts`: a per-process collector. It uses `performance.eventLoopUtilization()`, `monitorEventLoopDelay` (20 ms), `process.cpuUsage()`, the main thread's `/proc/self/task/<pid>/stat` on Linux, and six 10 s buckets of native `perf_hooks` histograms. One unref'd timer rotates the buckets. Nothing on the hot path awaits, allocates per call or does I/O. - `server/src/routes/instance-server-metrics.ts`: `GET /api/instance/server-metrics`, guarded by the existing `assertInstanceAdmin` (the check that `/instance/database-backups` and other instance routes use). It returns numbers, a random per-process `processId`, `uptimeSec`, `buildRevision` and `windowMs`. It returns no SQL, names, IDs or hostnames. - `server/src/app.ts`: when the flag is on, two middlewares time agent API requests from arrival to the API router, the route is mounted, and shutdown stops the collector. When the flag is off, nothing is mounted. - The snapshot also carries `dedicatedConnections` from `dedicatedDbConnectionStats()` (#21693): `limit`, `inUse`, `waiting` at request time and `peakInUse`, `queued`, `overLimit`, `nestedOverLimit` since start. The harness ignores it today; it shows dedicated-connection queueing in a restart burst. - `packages/db/src/statement-count.ts` and `lazy-transaction-queries.ts`: a Drizzle logger that increments one counter per statement replaces Drizzle's no-op logger. No edit to `client.ts`. - `paperclip-control-plane-port.ts`: one call records runner `emittedAt` to committed row for each committed (not duplicate) runner event. - `native-execution-lease.ts`: one call counts a lease whose renewal deadline passed. - Harness: missed lease renewals are summed per `processId` (several processes, or a restart, no longer cancel out). A metrics URL on the target origin gets the board token in the `Authorization` header; `PAPERCLIP_SCALE_METRICS_TOKEN` still overrides it, and another origin gets no token. - Contract: `scripts/scale-ramp/server-metrics.fixture.json` is the serializer's own output. A server test asserts the serializer produces exactly this file, and a harness test parses it and checks every field. - `rules/company-scope-exceptions.mdc`: adds the snapshot as an instance-wide exception with its reason. The runbook in the plan doc documents the path, the flag, the credential and each field. | Field | Provided | Source | | --- | --- | --- | | `mainThreadCpu` | yes (Linux; null elsewhere) | `/proc/self/task/<pid>/stat` utime+stime | | `eventLoopUtilization` | yes | `performance.eventLoopUtilization()` | | `eventLoopDelayP99Ms` | yes | `monitorEventLoopDelay`, highest 10 s p99 in the window | | `statementsPerSecond` | yes | Drizzle logger counter (not BEGIN/COMMIT or raw driver calls) | | `poolWaitP99Ms` | absent | postgres.js has no cheap hook for queue time | | `runEventLatenessP99Ms` | yes (null with no events) | runner event append | | `agentApiPreHandlerP99Ms` | yes (null with no agent requests) | arrival and router-entry middlewares | | `leaseRenewalsMissed` | yes, counter since process start | native lease deadline timer | ## Verification - `cd server && npx vitest run src/__tests__/instance-server-metrics-routes.test.ts src/__tests__/server-metrics.test.ts`: through `createApp` on the embedded test database. 200 with the exact field set for an instance admin; 403 for a company owner who is not an instance admin, for an agent key, and unauthenticated; 404 and no collector when the flag is off; values move after agent requests, statements, a late run event and a missed renewal; the body has no IDs, keys or SQL. - `cd server && npx vitest run src/services/native-runtime/native-execution-lease.test.ts src/services/native-runtime/paperclip-control-plane-port.test.ts src/__tests__/api-server-timing.test.ts`: the lease and run-event hooks (red/green checked for the lease hook by removing the call). - `cd packages/db && npx vitest run src/lazy-transaction-queries.test.ts src/client-options.test.ts`: the counter counts statements inside transactions and savepoints. - `node --test scripts/scale-ramp/scale-ramp.test.ts` and `tsc -p scripts/scale-ramp/tsconfig.json`: 38 pass, including the fixture contract and the board-token header. - `node server/scripts/typecheck-layers.mjs` and `tsc --noEmit` in `packages/db`: pass. Overhead, measured on a shared, loaded laptop (load average 8 to 20), median with min and max: | Measure | Off | On | | --- | --- | --- | | Logger call per statement (5M calls x 7) | 0.28 ns (0.24 to 0.97) | 3.36 ns (3.32 to 6.16) | | CPU per `select 1` statement, embedded Postgres (3,000 x 10) | 28.7 us (20.3 to 42.4) | 32.3 us (20.7 to 45.5) | | CPU per HTTP request, client and server in one process (2,000 x 6) | 67.0 us (40.4 to 107.4) | 64.2 us (40.9 to 83.3) | | Idle process CPU, 5 s x 3 | 0.01% of a core (0.00 to 2.21) | 0.08% (0.05 to 0.09) | | Record one run event (Date.parse + histogram) | - | 148 ns (135 to 188) | | One snapshot | - | 55 us | The statement and request differences are inside the run-to-run spread. The direct cost is about 3 ns per statement, against about 30 us of CPU per statement. ## Risks - Low risk. Off by default; off, only the statement counter runs, and it costs one increment more than Drizzle's no-op logger. - `mainThreadCpu` assumes Linux USER_HZ of 100 and is null on other platforms. The harness then uses `eventLoopUtilization`. - Run event lateness compares the runner's clock with the server's clock, so clock skew adds to it. Negative values count as zero. - With several processes, each request reaches one process. The harness keeps the highest gauge and sums counter growth per process, but a process it never samples is not seen. - The statement counter does not count BEGIN/COMMIT or raw postgres.js calls. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits, local test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
328a954cc1 |
test(scaling): add a real-agent ramp harness for hosted instances (#21695)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The control plane must run 1,000 agents at once on one workspace (`doc/plans/2026-11-13-thousand-concurrent-agents.md`) > - Above about 50 agents, the evidence is a load model only. Nobody has run 200 or 1,000 real agents > - Phase 3 of the plan proves the finish line with real runs on hosted instances, in a fixed order > - This pull request adds the harness for those runs. It does not run it against any real instance > - The benefit is that each proof step is one command with a written cost estimate, explicit pass/fail criteria, and a guaranteed cleanup ## Linked Issues or Issue Description **What problem does this solve?** The 1,000-agent plan needs real runs at 50, 200, 500 and 1,000 agents. No tool exists that creates real agents and tasks on a hosted instance, ramps concurrency, measures the control plane, and judges the plan's finish line. **Proposed solution** A script under `scripts/` that ramps real agents through the product API, stops at the first failed criterion, writes a machine-readable result, and always cleans up what it created. **Alternatives considered** Extend the restart-chaos hosted mode. That mode grades release swaps at up to 30 concurrent tasks. The ramp reuses its target guard and its cleanup order, but it has a different workload, different measures and a different stop rule. ## What Changed - `scripts/scale-agents-ramp.ts` and `scripts/scale-ramp/`: the harness. It reads the base URL from a flag or the environment and the credentials from the environment only. It creates or reuses a company, then creates agents, tasks, a project and an optional provider secret. It ramps concurrency in steps (default 10, 50, 100, 200, 500, 1,000) and keeps each agent on one small deterministic task at a time. - It measures per step: tasks started, finished, failed and needing Retry; failed runs; peak running agents; queue wait; time to first token and first activity (from sampled run events); run duration; board GET latency. An optional metrics endpoint supplies event-loop, statement and pool figures. A missing source is reported as "not measured", never as zero. - It evaluates the criteria from the plan's finish line plus harness criteria, writes them into `result.json` and `SUMMARY.md`, and stops at the first failure. - Cleanup runs in a `finally`: pause agents, cancel live runs, archive tasks, terminate and delete agents, delete the stored secret, archive the project and the created company, and wait for sandbox leases. `--cleanup-only <run id>` finishes a cleanup after a crash. `--dry-run` prints the plan and the estimate and makes no request. - Target guard: `--target-kind` is required and must match the host. Shared staging and production hosts need a second explicit flag. The script asks for the host name before it starts unless `--yes` is given. A run above the spend threshold needs `--confirm-spend <max USD>`. - `scripts/scale-ramp/scale-ramp.test.ts`: 28 tests against an in-process fake API with a virtual clock. They cover ramp logic, stop on failure, criteria evaluation, cleanup after a mid-run crash and an interrupt, `--cleanup-only` with and without a state file, dry run, the production-host guard, the spend gate, and that credentials never appear in output or files. - `scripts/scale-ramp/tsconfig.json` is added to the test programs that CI typechecks. The test file joins the `test:release-registry` node test list, which CI runs. - `doc/plans/2026-11-13-thousand-concurrent-agents.md`: a Phase 3 runbook with commands, cost, criteria, proof order, target prerequisites and open risks. ## Verification - `node scripts/ci-profile/node-test.mjs scripts/scale-ramp/scale-ramp.test.ts scripts/typecheck-test-programs.test.mjs`: 47 tests pass. - `node node_modules/typescript/bin/tsc -p scripts/scale-ramp/tsconfig.json`: no errors. - `node cli/node_modules/tsx/dist/cli.mjs scripts/scale-agents-ramp.ts --base-url https://scale1.test.paperclip.app --target-kind test-stack --steps 50 --hold-minutes 30 --dry-run` prints the plan, the estimate and the criteria, and makes no request. - The harness was not run against any real instance. That needs the owner's go-ahead. ## Risks - Low risk for the product: the change adds a script, tests and docs only. No server code changes. - A real run creates real agents, tasks and sandboxes and spends money. The guard, the typed confirmation, the spend gate, the task cap and the recorded-spend stop limit this. The runbook lists the open risks. - The cost defaults are assumptions until a first real run records actual spend. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
c66530da63 |
fix(server): release settled native run process identity at graceful shutdown (#21689)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A server process records the local process of each native run in
`heartbeat_runs` (`process_pid`, `process_group_id`,
`process_started_at`, `process_host_hash`)
> - Only a server in the same PID space can prove that such a process
stopped. `proveLocalOwnerGone` does not accept a recorded process group
from another container
> - Each deploy replaces the server process. Settled native runs that
the old server ran keep their process fields, and no later server can
release them. Recovery sweeps had to learn to skip these rows one by one
> - The stopping server is the one process that can check its own PID
space. After its graceful drain, it can see that a settled run's process
exited
> - This pull request adds a graceful-shutdown step that releases the
process identity of those runs, with the same rules that the running
server already uses, in one candidate read and one update
> - The benefit is that a deploy no longer adds stranded process records
for the runs that the stopping server could prove stopped, and a
successor on the same host has nothing to release at startup
## Linked Issues or Issue Description
Refs #21590
**What happened?**
After a graceful shutdown, settled native runs (`failed`, `cancelled`,
`timed_out`, `interrupted`) that the stopping server ran keep
`process_pid` and `process_group_id`. The successor runs in a new
container with a different `process_host_hash`. It cannot prove that the
recorded process stopped, so the rows stay forever. A successor on the
same host releases them at startup with about 10 statements for each
run.
**Expected behavior**
The server that ran the process knows that the process exited. A
graceful shutdown should release these records before the server stops.
An abrupt stop should keep today's behavior.
**Steps to reproduce**
1. Run native work so that some runs settle while the server runs, or
settle during the shutdown drain.
2. Send SIGTERM to the server.
3. Count settled native runs with `process_pid` or `process_group_id`
set: the count does not go down after the deploy.
**Paperclip version or commit**
master at
|
||
|
|
3eabdd6d10 |
perf(server): size run-log uploads and object journal IO from open run logs (#21685)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - One server process controls every run of a workspace, and the target
is 1,000 concurrent agents on one server
> - Each native run writes its run log to object storage. The server
batches a run's log lines and uploads one batch per run each second
> - The server allowed only 4 log uploads and 8 object storage
operations at a time, for the whole process. These numbers were set when
50 agents was a lot
> - At 200 runs, the run logs fell about 8 seconds behind. At 1,000
runs, pending log bytes grow toward the 64 MiB process cap. Past that
cap, run events wait and log appends can fail
> - This pull request sizes the log upload slots and the object storage
operation slots from the number of open run logs, between the old floor
and a ceiling that comes from the storage client's socket budget
> - The benefit is that run logs stay about one flush interval behind at
200 runs, and fall 7.7 seconds behind instead of 44 seconds at 1,000
runs (p50), without more sockets than the storage client allows
## Linked Issues or Issue Description
Refs plan item 1g in
`doc/plans/2026-11-13-thousand-concurrent-agents.md` ("Size the shared
limiters ... from the active run count").
**What existing behavior does this improve?**
The shared run-log upload limiter and the shared object journal
operation limiter.
**Subsystem affected**
Server: object storage run logs (`object-run-log-store.ts`,
`object-byte-journal.ts`) and the S3 storage client.
**Current behavior**
The process allows 4 run-log uploads and 8 object journal operations at
a time, for any number of runs. Each upload is 3 to 4 sequential object
writes. At 200 runs and 50 ms per write, the run logs commit with a p50
delay of 8.0 s, and only 55% of the log lines offered in a 20 s window
commit inside it.
**Proposed behavior**
The upload slots are one per 10 open run logs, at least 4 and at most
21. The operation slots are the upload slots plus 4 for reads, at most
25. That is half of the 50 sockets that the S3 client may open, so
downloads and other storage calls keep the other half. The S3 client now
sets its 50-socket limit explicitly, from the same constant.
**Reason and benefit**
With the change, 200 runs commit their logs at the flush interval (p50
1.1 s), and 1,000 runs commit with a p50 delay of 7.7 s instead of 43.8
s. Fewer pending bytes stay in memory, so the 64 MiB pending cap and its
admission failure are farther away.
**Breaking changes**
None. With 40 or fewer open run logs the limits are the same as before
(4 uploads, 8 operations).
## What Changed
- `server/src/lib/active-run-slots.ts`: a small helper that gives a slot
count from an active run count, clamped between a floor and a ceiling.
- `server/src/storage/s3-client-limits.ts` and `s3-provider.ts`: the S3
client sets `maxSockets` to 50 (the SDK default) from a named constant,
so the budget is explicit.
- `object-byte-journal.ts`: the object operation limit is the upload
slots plus a read reserve of 4, from the number of open run-log writers.
Open writers are counted with `trackObjectJournalWriter()`. When the
limit grows, waiting operations start at once. When it shrinks, running
operations finish and no new ones start until the count is below the
limit.
- `object-byte-journal.ts` also owns the upload limiter
(`acquireObjectJournalUpload`), process-wide, next to the writer count.
When a run log opens, waiting uploads start at once. When run logs
close, running uploads finish and no new upload starts until the running
count is below the new limit.
- `object-run-log-store.ts`: uses the shared upload limiter. A run log
registers when it begins and releases when it is finalized. The waiting
queue stays first in, first out, and each run has at most one upload in
flight, so one run cannot starve another.
- Review fix: the first revision kept the upload limiter per store, so a
run log opened by another store did not wake uploads waiting in this one
(a lost wakeup on growth). The shared limiter fixes it; a new test
covers growth and shrinkage with queued work.
- Tests: limit growth with uploads waiting and shrinkage with uploads
running; helper bounds; the storage socket budget; 200 open runs upload
at least 20 logs at once (4 before); 1,000 open runs stay at the ceiling
and every upload finishes.
## Verification
- `cd server && npx vitest run src/lib/active-run-slots.test.ts
src/services/object-run-log-store.test.ts
src/services/object-byte-journal.test.ts`: all pass.
- Red/green: with the slot count forced back to the fixed floor of 4,
the two new scale tests fail (4 uploads in flight); with the change they
pass.
- Simulation (in-memory object store with 50 ms per write, each run
appends 1.5 KB at 0.7 records/s, the recorded native event rate, 20 s
window, flush every 1 s). Counts are stable on a loaded machine because
the latency is simulated:
| Runs | Build | Peak concurrent writes | Writes/s | Log lines committed
in window / offered | Commit delay p50 / p99 |
| --- | --- | --- | --- | --- | --- |
| 200 | before | 4 | 74 | 1,473 / 2,669 | 8.0 s / 13.5 s |
| 200 | after | 20 | 211 | 2,646 / 2,790 | 1.1 s / 1.3 s |
| 1,000 | before | 4 | 75 | 1,229 / 13,499 | 43.8 s / 62.3 s |
| 1,000 | after | 21 | 392 | 7,789 / 13,667 | 7.7 s / 12.6 s |
## Risks
- More object storage writes per second at high run counts (at 1,000
runs and 50 ms per write, about 390/s instead of about 75/s). This
raises object storage request cost in proportion. The ceiling bounds it.
- The S3 client now states its socket limit. The value is the SDK
default, so behavior does not change.
- The open-writer count is process-wide. A run log that is never
finalized keeps its count, as it already keeps its writer entry. The
ceiling bounds the effect.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test runs).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
|
||
|
|
e8c79235ca |
perf(attention): scope run lifecycle retention hints and read only used columns (#21579)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The `attention-retention` queue archives old attention items (failed runs, reviews, questions, recovery actions, blocked tasks). A company hint makes it reload every candidate of the company, 100 per page, with eligibility reads for each page. > - Several run lifecycle producers sent that company hint: every agent status change after a run, the execution-control writes and legacy run terminalization when they name only a run, scheduled retries, and the Runner's task monitor tool. > - Each page also read whole rows it did not use: full `heartbeat_runs` rows (recent ones carry tens of KB of `context_snapshot` and `runner_profile_json`), full `issues` rows (descriptions), and the whole agent roster with its configuration. > - This pull request names the task graph or the source kind in those hints, and reads only the columns each eligibility decision uses. > - The benefit is fewer company reloads per run event and far fewer result bytes per reconstruction, with the same decisions. ## Linked Issues or Issue Description No issue exists. This describes the problem with the fields of `.github/ISSUE_TEMPLATE/enhancement.yml`. Stacked on #21552 (it builds on that pull request's change to `execution-control-notifications.ts`); review that one first. **What existing behavior does this improve?** The attention retention queue (`server/src/services/attention-retention.ts`) and the producers that wake it on run lifecycle paths. **Subsystem affected** Server background retention (`attention-retention.ts`, `retention-work-notifications.ts`, `execution-control-notifications.ts`, `heartbeat/agent-status.ts`, `heartbeat/retries.ts`, `native-runtime/paperclip-runner-tool-authority.ts`). **Current behavior** - `heartbeat/agent-status.ts` sends `{ companyId }` on every agent status change after a run (two places). `heartbeat/retries.ts` sends it for each scheduled retry. `writeExecutionControl` and `notifyExecutionRunWork` send it when the scope names only a run. The Runner's `set_task_monitor` tool sends it for one task. - On a local fixture with 1,500 open tasks and run churn, the queue loaded 47 company candidate rows per second from these hints. - `pageEvidence` read `select *` from `heartbeat_runs`, `issue_recovery_actions` and (through `getTableColumns`) `issues`; `reviewEligibility` read `select *` from `issues`; `interactionEligibility` read the full agent roster; the per-entry `check` repeated full-row reads. **Proposed behavior** - Agent status changes send `{ companyId, sourceKind: "agent_error_alert" }`. The status moves between `running`, `idle` and `error`. All three are invokable for blocker attention (`BLOCKER_ATTENTION_INVOKABLE_AGENT_STATUSES`) and for questions (`agent-eligibility.ts`). Failed-run attention only excludes terminated agents, and this path never sets that. So only agent error alerts can change. - Run-scoped execution-control writes, legacy run terminalization and scheduled retries use `notifyIssueAttentionRetentionWork` with the run id (the run's task, its parents and the tasks it blocks; company fallback past 100). This is the same helper as #21521. The task monitor tool scopes to its task. - `notifyIssueAttentionRetentionWork` matched the run's saved task id with `issue.id::text = …`. It now validates and casts the id, so the primary key is used, as #21552 does for recovery. - Narrow reads, same decisions: - runs: `id`, `agent_id`, `created_at` and the saved task id computed in SQL with the same rules as `readRunIssueId` (`issueId` unless it is absent or JSON null, then `taskId`; only non-empty strings). - recovery actions: `id`, `source_issue_id`. - issues for evidence and `visibleIssue`: the blocker-attention input columns plus `unblock_descriptor` and `blocked_transition_at`. - review issues: `id`, `company_id`, `status`, `execution_state`, `assignee_user_id`. - agent roster for question addressees: `id`, `company_id`, `name`, `reports_to`, `status` (the `AgentOrgRow` that `evaluateAgentInvokability` takes). **Reason and benefit** A run event no longer reloads every attention candidate of the company. A reconstruction no longer transfers whole run, task and agent rows. ## What Changed - `server/src/services/attention-retention.ts`: column lists for runs, recovery actions, issues and the roster; the run probe type replaces whole rows. - `server/src/services/heartbeat/agent-status.ts`, `heartbeat/retries.ts`, `execution-control-notifications.ts`, `native-runtime/paperclip-runner-tool-authority.ts`: scoped attention hints. - `server/src/services/retention-work-notifications.ts`: indexed lookup of a run's saved task. - `server/src/__tests__/attention-retention-work.test.ts`: two tests. ## Verification - `reads only the fields its decisions use from large runs, reviews and agents` (I saw it fail first): 50 failed runs, 50 tasks in review with large descriptions, 10 questions to a paused agent, about 40 KB of payload in each run's snapshot, profile and stdout and in the agents' configuration. It loads the three kinds with the full-size response capture (`full-database-response-capture.ts`). Before: `heartbeat_runs` 6,317,700 bytes, `agents` 84,336 bytes, `issues` 4,235,810 bytes. After: under the test bounds of 20,000, 5,000 and 60,000 bytes. The decisions are the same before and after: 50 failed runs, 40 reviews and 10 questions. - `scopes run lifecycle attention hints to the run's task graph and agent status to agent alerts` (I saw it fail first): `writeExecutionControl` and `notifyExecutionRunWork` with a run id send the hint for the task and its parent. `finalizeAgentStatus` sends the agent-alert hint. Before, all three sent `{ companyId }`. - `attention-retention-work` (57 tests), `attention-retention-candidates`, `attention-service`, `execution-control-work` and `heartbeat-agent-status-recovery` pass, each file run alone (in `server/`, against a dedicated local `_test` database). - `pnpm check:module-boundaries` passes. Not done in this pull request, and why: - I did not reproduce `syncItems` re-syncing unchanged rows from the retention `check`. On my fixture (300 stalled blocked tasks, 4,500 old recovery-action and blocker retention rows, run churn), `syncItems` ran 0 times in a 180-second window and `decision_retention` reads were 0 per second. `check` calls `syncItems` only for entries that are already due (older than the 90-day archive delay), and a due entry is archived and leaves the candidates. So I did not batch `syncItems` or skip checks of unchanged rows. - `readIssues` still selects the `issues` columns that blocker attention needs. I did not narrow `issueService.listBlockerAttentionBatch`. ## Risks - A producer that changes more than the run's task graph would now miss the wider effect. Company-level changes (agent configuration, company status, budgets) keep their company hints. The 90-day archive delay makes a missed reload cheap: the item is reloaded on the next hint that covers it. - The saved task id now comes from SQL. The SQL mirrors `readRunIssueId` for strings, absent keys and JSON null. A non-string `issueId` value now yields no task, as before. - No schema change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [ ] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
bbed32450c |
perf(load-model): model recorded native run rates, the recorded event mix and coalesced delta batches (#21549)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - `scripts/benchmark-agent-scaling-load.ts` (#21418) estimates how many concurrent agents one server process can carry > - It assumed 20 text deltas/s and 1 other event/s per agent, and every other event was an `item.completed` > - Recorded hosted native runs show about 0.2 deltas/s and 0.5 other events/s, with a mix in which 39% of the other events are server trace spans that take a cheaper path > - This pull request makes the recorded rates and mix the model's default, keeps the old rates as an explicit stress profile, adds a burst profile, and mirrors the delta batching of #21520 > - The benefit is a model that measures what production does, so the next optimizations target the real per-agent cost ## Linked Issues or Issue Description Refs #21418 (load model and plan). Refs #21520 (delta batching that the model now mirrors). **What existing behavior does this improve?** The load model's default rates, its event mix and its delta batching rule, and the plan document's measurements. **Subsystem affected** `scripts/benchmark-agent-scaling-load.ts`, its test `server/src/__tests__/agent-scaling-load-model.test.ts`, and `doc/plans/2026-11-13-thousand-concurrent-agents.md`. **Current behavior** The model defaults to 20 deltas/s and 1 other event/s per agent. Every other event is a native `item.completed` with a chat-progress wake and a checkpoint save. It cuts a delta batch on the timer even while the previous batch commits. **Proposed behavior** - `PROFILE=recorded` (default): 0.2 deltas/s and 0.5 other events/s per agent. `PROFILE=burst`: 2.5 and 3, a run's busiest minute. `PROFILE=stress`: the earlier 20 and 1. API requests stay at 0.3/s. Each rate can still be set on its own. - `EVENT_MIX=recorded` (default) sends other events in the recorded proportions. A `run.performance.span` is an ordinary append with no wake and no checkpoint, as the server's run trace writes it. `tool.execution.*` and `item.completed` are native appends with a chat-progress wake, then a checkpoint save. `EVENT_MIX=<type>` sends one type only. - Delta batching follows #21520: deltas that arrive while a batch commits all join one next batch. - The report adds `profile` and `eventsIssuedByType`. **Reason and benefit** At the recorded rates the per-agent cost is the non-delta runner events and their checkpoint saves, not deltas. The plan and the next optimizations need that picture. **Breaking changes** None for the product. Anyone who runs the model with no options now gets the recorded profile; `PROFILE=stress` gives the earlier load. ## What Changed - Profiles, the recorded event mix (a 100-slot cycle per agent, offset by agent so the fleet sends the mix at every moment), the span append path, and the coalesced delta batches. - The model test now checks the profile in the report and that both server trace spans and runner events with a progress wake are in the mix, for the recorded profile and for the stress profile with one transaction per delta. - The plan has a new section "Load model at recorded rates": the recorded source (40 recent native runs on one hosted workspace, as aggregates), the profiles, the mix and its server paths, fresh runs at 500, 1,000 and 2,000 agents plus burst and stress, and what they show. It records the 3-connection checkpoint pool as a computed per-process limit on a hosted database, and that SQL building in Drizzle costs about as much main-thread time as result parsing on the hosted instance. A new section "Considered and not done" records why `run.performance.span` rows stay (about 3% of the statements per agent, with readers) and why checkpoint saves per critical event only are not done now. ## Verification - `server/src/__tests__/agent-scaling-load-model.test.ts` passes (2 tests). The new assertions fail against the previous script: it reports no `profile` and no `eventsIssuedByType`. - `server/src/__tests__/agent-scaling-load-model.test.ts` also checks each profile's rates with no overrides, and that a rate set on its own wins. - Load model runs, recorded in the plan (60 s, main pool of 10, embedded PostgreSQL on the same machine, each run started below a 1-minute load average of 15): recorded rates at 1,000 agents (event-loop utilization 0.61) and 2,000 (0.85) finish every operation with no failure; 500 agents all in their busiest minute run at 0.95 with p99 lateness under 0.4 s. About 4.4 statements per agent per second at recorded rates, 35% of them on the checkpoint pool. - Earlier runs at load averages of 20–55 had shown one process falling behind at 500 burst agents and at 2,000 recorded; the quieter runs after the model fixes (event order, spans off the runner stream) do not, and the plan says so. The checkpoint pool as a per-process limit is now stated as a computed limit for a hosted round trip (about 400 agents in a busy minute at 1 ms), not as measured. - Greptile findings fixed: a waiting delta batch no longer takes deltas sent after the next runner event, takes every delta queued when it starts, and spans no longer join the runner event stream. ## Risks - Low: a benchmark script, its test and a plan document. No product code changes. - The machine that measured the plan's table was shared; the plan marks the rows that the machine distorted. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
a4c570aede |
feat(mcp): ask an app's form questions in the task chat and send the answer back (MCP elicitation) (#21466)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Agents use connected MCP apps through the tool gateway. > - An MCP app can ask the person a question in the middle of a tool call (MCP elicitation). There are two kinds: a form, or a link to open. > - Today a form question is a dead end: > - A 2026-07-28 server's form `input_required` stops with "needs information Paperclip can't collect yet". > - An older server's `elicitation/create` makes a card, but the answer is never sent back, and the server waits on an open stream. > - Paperclip does not declare form support, so compliant servers never ask. > - This pull request shows the question as a card in the task chat. The card names the app, and only a person can answer it. When the person answers, the agent wakes. The agent's repeated call sends the answer to the app exactly once, for both protocol eras. > - The benefit: apps that need a choice or a detail from the person now finish their work. Restarts lose nothing, and no call is left waiting. ## Linked Issues or Issue Description Refs #21140 (stateless MCP client; this PR closes its "form elicitation" follow-up). Refs #21263 (MCP Tasks; this PR shares its card code). Refs #21389 (MCP Tasks follow-ups: the card answer validation item). **Is your feature request related to a problem?** When a connected MCP app asks the person a form question during a tool call, the call stops and cannot continue. A 2026-07-28 server gets a "needs input" error. An older server gets a card whose answer never reaches it. **Describe the solution you'd like** Show the question in the task chat. Let a person answer, skip, or leave it. Wake the agent when the question is resolved. Send the answer on the agent's repeated call, once, with the server's exact state. **Describe alternatives you've considered** Holding the tool call open until the person answers. We rejected this: it holds a stream and a run slot, and it does not survive deploys or restarts. The plan (`doc/plans/2026-11-13-mcp-elicitation.md`) records nine design decisions with their alternatives. ## What Changed - **Plan:** `doc/plans/2026-11-13-mcp-elicitation.md` covers the spec, the inventory, the decisions, the journey, tests, and risks. - **New module `server/src/services/mcp-elicitation.ts`.** It is the single mapping for direct calls and MCP Tasks: - It turns each spec form field into a real card field: text with defaults, number, yes/no, single choice (`enum` and `oneOf`), and multiple choice (`items.enum` and `items.anyOf`). - It turns card answers back into typed `content`. - Answered sends `accept`. Skip sends `decline`. Withdrawn, closed, or expired sends `cancel`. - It refuses forms that ask for passwords, keys, tokens, card numbers, or one-time codes. - It keeps app text plain, so links in it are not clickable. - It seals `requestState` with AES-256-GCM, bound to the company and connection. - **New table `tool_mcp_elicitations`** (migration 0479). It is company-scoped, with one row per question round. It holds a keyed call identity, the sealed state, the questions and their cards, a claim time, and a 24-hour expiry. - **Gateway, stateless servers:** - Paperclip declares `elicitation: {form, url}` on task tool calls. - A form-only `input_required` saves the round, makes the cards, and stops with a wait message. - A repeated call before the person answers stops again without contacting the app. - After the answer, the repeated call claims the round and sends `inputResponses` and the exact `requestState`, once. - A failed send releases the claim. A crash mid-send lets the claim lapse. - **Gateway, older servers:** - Paperclip declares form support at `initialize`. - The first question gets `cancel` at once, so the server never waits. - When the same question (same words and schema) arrives on the repeated call, Paperclip answers it from the card. - **Gateway, mixed rounds and retries:** - A form beside a URL step shows the link first. - A repeated call that stopped for a question runs again in the same run. It is no longer replayed by the per-run idempotency key. - **Cards:** the title is "*App* needs your input". The resolver policy is `human_only`. "Send to *App*" sends the answer. A user comment does not supersede the card. - **Activity log:** new events `tool_gateway.elicitation_sent`, `tool_gateway.elicitation_refused`, and `tool_gateway.elicitation_expired`. They record the action only, never the values. - **MCP Tasks:** task cards use the same mapping and gain the app name, real fields, and `human_only`. A secret or an unsupported form stays unsupported. - **Storybook:** `Tasks and conversations/Questions and approvals/MCP App Questions` has five states: Asking, Answered, Skipped, Expired, and TwoQuestions. The fixtures are generated from the server mapper's output. - **Docs:** `doc/connections/MCP-ELICITATION.md` (linked from the README) is new. `MCP-PROTOCOL-VERSIONS.md` and `MCP-TASKS.md` are updated. ## Verification - `server/src/__tests__/mcp-elicitation.test.ts` has 16 tests. They pass on the branch. - **Red/green:** 9 of the 9 gateway tests fail with master's `tool-gateway.ts` and `mcp-http.ts`. The 7 pure unit tests cover the new module. - The tests use a fake 2026-07-28 server and a fake 2025-06-18 server that asks over the open stream and waits for the reply. - Covered: answer, skip (`decline`), withdraw (`cancel`), two questions at once, a restart while waiting, a crash mid-send, another company with the same app and arguments, expiry, a refused password field, an agent that tries to answer (rejected), and capabilities. - Existing suites: `mcp-tasks`, `mcp-protocol-versions`, `tool-gateway-service`, `tool-gateway`, and `remote-mcp-connectors`. See the results comment below. - Storybook: the five stories render in headless Chromium, and the play tests pass. The only console errors are the 404s from the backend-less API calls that every story makes. - `tsc --noEmit` is clean for `server` and `ui`. `node scripts/storybook-catalog.mjs` passes. ## Risks - **Older servers may start asking questions.** Declaring form support at `initialize` can make older servers ask where they did not before. Mitigation: the first question always gets `cancel` at once, so no call hangs, and the agent gets a clear next step. - **Per-call cost.** Each 2026-07-28 HTTP tool call in a task makes one indexed lookup for a waiting round. Nothing is added before the first token or to agent boot. - **Migration.** One additive table with no backfill, so it is safe to apply online. - **Changed Tasks behavior.** A task card that is withdrawn or closed now sends `cancel` instead of `decline`, as the spec defines. A skipped card still sends `decline`. - **Stale cards.** Expiry is checked only when the agent calls again. A card can sit past 24 hours until the next call. Then it is withdrawn and the app asks again. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code with tool use (shell, file edits, headless browser), with extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
05fef9f27b |
perf(server): add a load model of per-agent server cost and the plan for 1,000 concurrent agents (#21418)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - One server process controls every agent run of a workspace > - A hosted workspace slows down with about 50 agents running at the same time > - The target is 1,000 agents at the same time, served by several server processes > - This pull request adds the plan and a local load model that measures what each running agent costs the server > - The benefit is a measured ranking of the causes, and a repeatable before/after number for each fix ## Linked Issues or Issue Description **What existing behavior does this improve?** The server controls agent runs. With about 50 runs active, requests wait and run events fall behind. **Current behavior** Each text delta from an agent is one database transaction. In the load model, 50 agents at 20 deltas per second issue about 3,900 statements per second, and the process completes about half of the offered deltas. Agent authentication slows from 7 ms to 464 ms because it shares the connection pool. **Proposed behavior** The plan gives three phases: measure, decrease the cost per agent, then let several server processes own runs. Each fix is one pull request with a test and a before/after number from the load model. **Reason and benefit** The ranking was an estimate from code. The load model replaces the estimate with a measurement and gives each fix a number to improve. ## What Changed - Added `doc/plans/2026-11-13-thousand-concurrent-agents.md`: finish line, evidence, load model result, phases, and owner decisions. It now also has a "Load model after Phase 1" section. That section has a table at 50, 100, 200 and 400 agents and a row at 200 agents with deltas and leases as before Phase 1. The 400-agent delta count was measured before the count fix and is marked as not reliable. - Added `scripts/benchmark-agent-scaling-load.ts`: N simulated agents call the real event append, controller checkpoint save, lease renewal, and agent authentication code against an embedded PostgreSQL. It reports statements for each database pool, event-loop utilization, and lateness for each source. - Deltas: by default the model commits deltas as the server port does after #21460. The first text after another event commits at once. Later deltas commit in one transaction (`appendHeartbeatRunEventBatch`) per 250 ms or 100 deltas, and the stream does not wait for the receipts. `DELTA_BATCH_MS=0` keeps one transaction per delta, as before batching. - Leases: by default the model renews every lease through the shared schedule from #21444 (`startNativeSessionExecutionLeaseRenewal`, `renewNativeExecutionLeasesNow`). `LEASE_MODE=single` keeps one transaction per run, as before. - The commit counter is read before the checkpoint drain, so commits that come after the window do not count in it. - Per-source counts freeze when the window closes. An operation that settles later, while queued work drains, is not done. - Batch lease mode needs `LEASE_S` of at most 20. The shared schedule renews a remote lease on its own 20 s after its last renewal, and the model does not count that work, so the script rejects a longer interval. - Added `server/src/__tests__/agent-scaling-load-model.test.ts`. It runs the model end to end in both modes: the default (batched deltas and shared lease renewal) and `DELTA_BATCH_MS=0 LEASE_MODE=single`. It checks that every source finishes without failures and that each pool is counted. ## Verification - `pnpm --filter @paperclipai/server exec vitest run src/__tests__/agent-scaling-load-model.test.ts` on a tree that contains #21460 and #21444: 2 of 2 pass (both modes). The test also checks that no source reports more done than offered. - `pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-agent-scaling-load.ts` with `AGENTS=50 SECONDS=20` and higher agent counts. The results are in the plan. - `pnpm check:module-boundaries` passes. - No product code changed. ## Risks Low risk. The pull request adds a document and a script that no product code imports. The load model runs on loopback PostgreSQL, so its absolute numbers are not a prediction for a hosted database. The plan states this limit. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), Claude Code, with tool use and code execution. Subagents on the same model read the code paths. - The two-mode load model test and the merge with master: Claude Opus 5.5 (`claude-opus-5-5`), Claude Code, with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [ ] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com> |
||
|
|
cfa959add7 |
refactor(slack): correct stale status comment and add production acceptance checklist (#20840)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The Slack chat connector lets people talk to their agents in Slack > - The connector is close to production, but its live evidence is out of date > - The September live results use old revisions, and the October setup run did not test follow-up, reauthorization, or removal > - Cloud managed setup has never run against real Slack, and its docs say not to enable it until it has > - This pull request adds one checklist for a fresh live pass on current master, and corrects one stale code comment > - The benefit is a clear finish line for Slack production readiness on self-hosted and Cloud ## Linked Issues or Issue Description **Issue type** Missing documentation. **Where is the issue?** `doc/connections/SLACK-AUTOMATIC-SETUP.md`, `doc/plans/chat-adapters/2026-09-05-slack-live-qualification-result.md`, `doc/testing/slack-cloud-qualification.md`, and `server/src/services/chat-channels.ts`. **What's wrong?** The Slack live qualification is spread over three documents. Their evidence uses older revisions. No document lists what a production pass must cover for Cloud managed setup. A code comment says the manifest does not request `assistant:write`, but the manifest does request it. **Suggested fix** Add one dated acceptance checklist with pass conditions per row. Correct the comment. ## What Changed - Added `doc/plans/2026-11-11-slack-production-acceptance.md`. It has sections for self-hosted automatic setup, manual and existing apps, conversation behavior, Cloud managed setup on staging, and distribution questions (Marketplace-only workspaces, non-Marketplace rate limits, scope justification, zero-copy). - Corrected the comment in `server/src/services/chat-channels.ts`. It now says that Slack shows the working state through `agents.sessions.setStatus` and the lifecycle reply. No code behavior changed. ## Verification - Documentation and a comment only. No runtime behavior changed. - I checked the doc links and the referenced files against current master. ## Risks - Low risk. The change is documentation and one comment. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and web search. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [ ] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
944b1c64d7 |
feat(mcp): keep long provider calls durable with the MCP Tasks extension (#21263)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Agents use connected apps through Paperclip's MCP client in the tool gateway > - Some provider actions run for minutes: CI runs, deploys, batch jobs > - Today such a call is lost if it outlives the tool timeout or if Paperclip restarts or deploys mid-call, and a retry can start the work a second time > - Recovery must just work, and the target rate of runs that need a manual retry is 0% > - This pull request implements the MCP Tasks extension (`io.modelcontextprotocol/tasks`, 2026-07-28) in the MCP client > - The benefit is that a long provider call becomes a durable task: it survives timeouts, restarts, and deploys, and it is never started twice ## Linked Issues or Issue Description **Problem** A long provider tool call holds an HTTP request open. When it passes the tool timeout, or when Paperclip restarts or deploys during the call, the result is lost. The agent then often repeats the call, and the provider does the work again. **Proposed solution** Implement the client side of the MCP Tasks extension. Declare it on stateless tool calls. Store each returned task durably, wait on it during the call, keep polling it in the background after the call ends, and let the same call attach to the stored task instead of sending `tools/call` again. **Why it matters** CI, deploy, and batch actions are where agents lose the most work today. A durable task handle makes these actions safe across timeouts, restarts, and deploys, without a person. Refs #21119, Refs #21140, Refs #21219 (the MCP Apps, stateless MCP client, and MCP Skills work this builds on). ## What Changed - **Off by default:** MCP Tasks are behind the new instance experimental setting `enableMcpTasks` ("Durable app actions (MCP Tasks)"), default off. When it is off, Paperclip never declares the extension, nothing attaches to or polls tasks, pending rows do not block idle sleep, and behavior matches master. A test covers the off state. - `mcp-http.ts`: the tasks extension is declared per request, only on HTTP `tools/call` and `tasks/*`. Discovery, listings, and stdio servers do not declare it. `tasks/*` requests put the task ID in `Mcp-Name`. - New `mcp-tasks.ts`: parses task results, bounds the server's timing hints, sends `tasks/get`, `tasks/update`, and `tasks/cancel`, and seals task IDs at rest (AES-256-GCM, bound to company and connection; a keyed hash gives uniqueness). - New company-scoped table `tool_mcp_tasks` (migration `0475`, online-compatible). It links the task to the call, run, task, connection, and grant. It stores no provider result. - `tool-gateway.ts`: - A returned task is recorded, then polled up to the normal tool timeout. Its result goes through the same validation, redaction, and recording as a direct result. - Past the timeout, or when the caller disconnects, the agent gets `provider_task_pending` ("call again with the same arguments"). The task stays durable. - A call with the same endpoint, tool, full arguments, grant, agent, task, and actor attaches to the undelivered task. A retry with the same idempotency key also attaches, and it can never start different work under that key. - Input: URL requests show the provider link (Paperclip never answers them). Form requests become a question card; only the person's answer is sent with `tasks/update`. Other requests stop with a clear "needs input" error. - Stopping the run sends `tasks/cancel` and marks the task cancelled. A caller that only disconnects keeps the task. - `failed` becomes a tool error with the provider's message. An unknown task or a passed TTL becomes `provider_task_expired`. Every task stops being tracked at its TTL, or after 7 days at most. - Audit entries: `tool_gateway.task_started`, `task_input_sent`, `task_cancelled`. - New `mcp-task-work.ts`, registered at startup with the shared deadline scheduler: it resumes polling pending tasks after a restart, cancels tasks whose task closed or run was stopped, and wakes the assignee once when a task settles after its run ended. - Docs: `doc/connections/MCP-TASKS.md`, linked from `doc/connections/README.md`. Background work (per `rules/sleep-aware-background-work.mdc`): trigger is a task row write or release (`mcp-tasks` queue) and one startup pass. The deadline is the earliest `next_poll_at`. Cadence is the server's `pollIntervalMs` within 5 s – 5 min, with backoff; 5 min while waiting for a person. It is quiet when no row has a next poll. This is the narrow external-source exception: providers give no wake Paperclip can receive, and polling stops with the task. ## Verification - `server/src/__tests__/mcp-tasks.test.ts`: 20 tests with a fake 2026-07-28 server that returns tasks only to a client that declares the extension and checks `Mcp-Name`. They cover: - polling to completion, with the result identical to a synchronous call; - timeout, then an attaching call with no second `tools/call`; - a keyed retry, and a reused key with different arguments (refused); - restart mid-task: a fresh gateway with no in-memory state polls the row, wakes the agent once, and the recovered run gets the result once with no second `tools/call`; - no wake while the run is still going; - URL input (link, never answered), and form input (question card, then `tasks/update` with the answer); - cancellation by the run, by closing the task, and no cancellation when only the caller leaves; - failure, an unknown task, and TTL expiry (ends in under 3 s, no hang); - another agent never attaches; - synchronous calls unchanged and older servers never see the extension. - Red/green: with attach disabled, the timeout, keyed-retry, and restart tests fail (3). With `tasks/cancel` disabled, both cancellation tests fail (2). Before `tools/call` declared the extension, 13 end-to-end tests failed. All pass with the change. - Existing suites pass: `mcp-protocol-versions`, `mcp-http`, `mcp-apps`, `mcp-skills`, `server-startup-feedback-export`, `tool-gateway`, `tool-gateway-discovery-http`, `remote-mcp-connectors` (342 tests). - `tsc --noEmit -p server` is clean. `pnpm --filter @paperclipai/db check:migrations` passes (the migration is classified online-compatible). Snapshots pruned to five. - Agent boot and time-to-first-token: no change in the boot or first-turn path. The only new per-call cost is one indexed lookup on stateless tool calls. - Not run: a live provider that returns tasks. No public MCP server that returns tasks was available without new credentials. ## Risks - Shipped off by default. Review findings that only affect the setting-on path after this merge are tracked as follow-ups. - Migration: a new table with indexes and foreign keys to `companies`, `tool_invocations`, and `tool_connections`. Online-compatible. - Behaviour change only for servers that return tasks, which they may do only to clients that declare the extension. Older servers and stdio servers are unchanged. - A task that settles while nobody is waiting stays attachable for 1 hour, so a deliberate repeat of the same call within that hour collects the earlier result instead of starting new work. - The wake after a background task settles uses the generic task wake; the agent must call the tool again to collect the result. - Follow-ups: `notifications/tasks` via `subscriptions/listen`; sampling and roots input requests; showing a long action's progress in the task chat. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use (shell, file edits, local test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
3914d0c94a |
test(chaos): add DF1, a sandbox disk-full fault on the final workspace save (#20810)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart chaos harness (`tests/restart-chaos/`) runs real agent
tasks and breaks the control plane on purpose, so failures show up
before staging users see them
> - A census of 48 hours of staging failures found runs whose result
save-back failed because the sandbox disk was full (class F). The
harness never filled a sandbox disk, so it missed that class
> - #20758 made a full-disk save end recovery at once with cause
`sandbox_disk_full` and a Retry. Nothing in chaos exercised that path
end to end
> - A real full disk cannot be made on demand. The owner chose a
test-only hook that makes the save report a full disk, not a small disk
> - This pull request adds the hook at the save's sandbox boundary and a
DF1 fault that arms it, clears it, and grades the result
> - The benefit is that every wave can now check the disk-full rest, the
Retry offer, and that Retry works after space is freed
## Linked Issues or Issue Description
Refs #20758 (product behaviour under test), #20678 (finalization hands a
full-disk copy-back to recovery)
**What is the problem?**
The chaos harness had no fault for a full sandbox disk during result
save-back. Staging saw this failure (census class F in
`doc/plans/2026-11-11-staging-failures-vs-chaos-coverage.md`), and chaos
could not reproduce it.
**What is the expected behaviour?**
A wave can schedule DF1. While DF1 is armed, the task's final workspace
saves fail as a full disk. When DF1 is cleared, saves succeed again. The
task must rest with `sandbox_disk_full` and a Retry, and that Retry must
finish the task.
**Why does it matter?**
Without a fault, a regression in the disk-full path (no Retry, a wrong
cause, or a Retry that does not work) reaches staging users before chaos
sees it.
## What Changed
- `server/src/services/sandbox-disk-full-fault.ts` (new, test-only):
`sandboxDiskFullFaultTarget` wraps the target of a final save only when
the harness armed that save. The capture commands then exit 1 with the
exact message the in-sandbox script writes on ENOSPC, and the provider
download fails with `No space left on device`. The gate follows
`@paperclipai/shared/crash-points`: `PAPERCLIP_DISK_FULL_FAULT_CONTROL`
is read once at import and ignored under `NODE_ENV=production`.
Disarmed, the hook is one comparison and returns the save's own target.
- `server/src/services/remote-workspaces.ts`: the capture call in
`saveRemoteWorkspace` passes its target through the hook. No other
product change. `workspace-recovery.ts` is not touched.
- `tests/restart-chaos/disk-full-faults.ts` (new): the DF1 kind, the
control-file seam (atomic writes), the rest evidence
(`activeRecoveryAction.evidence.systemRecovery.failure.cause.label`),
the `disk_full_fault` T check, and the DF1 R3 rule (one expected Retry
that must finish the task).
- Harness wiring: `--faults DF1` (refused without `--env daytona`, with
a storm, or with PV faults), no scheduled restart, arm at launch, clear
at the first rest just before the Retry click, receipts in
`<wave>/disk-full-faults.jsonl`, and ledger fields `diskFull: {
faultedSaves, restedDiskFull, retryOffered, retryWorked }`.
- Docs: a "Sandbox disk full (DF1)" section in
`tests/restart-chaos/README.md`, and the census plan marks DF1 as built.
## Verification
- `cd server && pnpm exec vitest run
src/services/sandbox-disk-full-fault.test.ts`: the gate is off by
default and under `NODE_ENV=production`, the save's own target comes
back when disarmed, only a listed final save is faulted, the product
capture turns it into the named disk-full failure, and clearing the file
disarms the next save.
- `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts
tests/restart-chaos/disk-full-faults.test.ts
tests/restart-chaos/grader.test.ts tests/restart-chaos/scheduler.test.ts
tests/restart-chaos/provider-faults.test.ts`: planning, the control
file, and grading.
- `node scripts/typecheck-test-projects.mjs tests/restart-chaos` and the
server `tsc --noEmit`: clean.
- Live Daytona waves (Codex, `--storage s3`, image content `96c4339f…`,
run through the two-slot queue):
- `w-20261111T131314-s9411` (B1): the hook was armed in the server but
no save reached it. B1 uses a local worktree synced into the sandbox,
not a remote workspace. Graded `not_applied` as designed. The harness
now requires `--storage s3` and the docs point at R1.
- `w-20261111T131941-s9412` (R1): 3 faulted saves. t01 rested `blocked`
with `sandbox_disk_full`, `user_action_required`, board owner, API
`canRetry: true`. t02 was cleared too early, while recovery still
checked its source; that recovery then saved and the task finished.
Fixed in
|
||
|
|
62b7733161 |
test(restart-chaos): long mid-tool K15 outages, Runner evidence, and TM1 stalls on Daytona (#20711)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The restart-chaos harness (`tests/restart-chaos/`) restarts or stalls the control plane while real agent tasks run. > - The staging census (#20679, `doc/plans/2026-11-11-staging-failures-vs-chaos-coverage.md`) found two open classes that the harness could not create. In class R, a deploy lands mid-turn, the server stays down for 4–7 minutes, and recovery then finds the remote Runner gone (`runner_remote_process_identity_unavailable`). In class L, the server stalls for about 100 s without a restart, and legacy runs that are still `preparing` end up waiting for a manual Retry. > - K15 restarted at once. No Daytona workload kept a tool running across the fault. TM1 was local-only with a fixed range. Nothing recorded what the remote Runner did while the controller was away. > - This pull request adds a K15 outage hold, a mid-tool workload, sandbox-side Runner evidence, and TM1 stalls on Daytona with a configurable hold. > - The benefit: these staging failure shapes can now be reproduced, and each wave records the Runner's own state. ## Linked Issues or Issue Description Refs #20679 **What happened?** On staging, two failure shapes had no matching chaos scenario: - Class R: a deploy landed mid-turn and the server stayed down for 4–7 minutes. Recovery then found the remote Runner gone. Nothing captured why the Runner exited. - Class L: a 100 s server stall without a restart lapsed every controller lease. Legacy `preparing` runs then waited for a manual Retry. **What did you expect?** The harness should create a long mid-turn deploy outage and a stall without a restart on Daytona. It should also save the sandbox-side Runner evidence. **Steps to reproduce** 1. Use the census report and the staging run records listed in it. 2. Compare them with the harness options before this change: K15 has no hold, TM1 is local-only, and there is no Daytona tool workload. ## What Changed - `--k15-hold-seconds <n|min-max>` (0–600). A K15 stops the server with SIGTERM, keeps it down for the seeded hold, then starts the next deploy generation. The range is drawn only for K15 cells, so existing seeds keep their plans. Storms use it too. - New opt-in workload `T1`. The agent runs one shell loop in the foreground for about 100 s, then saves `TICKS-<nonce>=50`. Its fault window, 35–80 s after launch, lands mid-tool. - New `daytona-runner-logs.ts`, a read-only probe of each wave sandbox through the Daytona toolbox. It runs only on sandboxes with the wave's labels, and only on `started` ones. It records every process, memory, disk, `dmesg` OOM lines, and the tails of `runnerd.stderr.log`, `runner-process.identity` and `runner-state.json`. It redacts known secrets. It runs right after a K15 stop, just before the next deploy starts, and at wave end. Output goes to `results/<wave>/daytona-runner/` and to `runner_logs_captured` in the timeline. The probe never stops, starts or deletes a sandbox, and a failed capture is recorded instead of thrown. - TM1 now runs on `--env daytona`, and `--tm1-hold-seconds <n|min-max>` (10–600) replaces its default 95–150 s range with the same single draw. TM2 and TM3 stay local. - Tests: `k15-long-hold.test.ts` and `tm1-stall.test.ts`. The README is updated. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 45 files and 531 tests pass. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: passes. - Daytona waves, all with T1 and K15: | Wave | Revision and image | Hold | Result | | --- | --- | --- | --- | | `w-20261111T015441-s9310` | master |
||
|
|
3fdcc2a28c |
fix(runner): resend an ACPX turn whose start was cut off before the prompt left (#20675)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native ACPX run (for example Claude) sends `turn.start` to its Runner. The Runner asks the ACPX sidecar to start the turn, and the sidecar sends the prompt to the provider > - A start can fail without a definite answer. A host freeze can outlast the sidecar request deadline, or the sidecar or Runner can die during the start > - The Runner then saved the session `closed`. Every same-run resume failed with `ACPX session is not open`, and the task needed Retry > - The Runner cannot tell from its own state whether the provider has the prompt. Resending could duplicate it. A continuation could lose it > - This pull request adds a durable record that answers that question: a sidecar prompt-dispatch journal, written before the prompt reaches the provider > - The benefit is that a start cut off before the prompt left resends the prompt exactly once, with no Retry. Every case the record cannot prove keeps today's behavior ## Linked Issues or Issue Description **What happened?** Local restart-chaos wave `w-20261110T185315-s9107`, task t05 (claude, workload C1, fault TM2). A follow-up run reopened the parked Claude session and sent `turn.start`. The sidecar first starts Claude and loads the conversation; the prompt goes out only after that. After 24 s without a reply, the harness froze the server, Runner, sidecar and Claude (SIGSTOP) for 33 s. On thaw the Runner's 30 s sidecar deadline had passed. It reaped the sidecar and saved the session `closed`. Three same-run resumes failed with `PRP command turn.start failed: … ACPX session is not open`, recovery stopped with `native_session_retry_exhausted`, and a person had to click Retry. **Expected behavior** When a durable record proves the prompt never reached the provider, the run resends it once by itself. When the record cannot prove it, the run behaves as today. **Steps to reproduce** 1. On master, run `cargo test --manifest-path packages/paperclip-runner/runner/Cargo.toml -p paperclip-runner-core --test acpx_park_preparation cut_turn_start`. 2. `a_start_cut_off_before_the_prompt_left_reopens_and_sends_the_prompt_once` fails: the session is `closed` and the next start fails with `ACPX session is not open`. Refs #20507, #20645, #20674 ## What Changed - New file contract in `protocol/provider-schemas/acpx-sidecar.schema.json`: `turnDispatchJournalFile` (`paperclip-turn-dispatch.jsonl`) and `turnDispatchJournalRecord` (`received` / `dispatched`). The generator emits the file name and schema into TypeScript and Rust, so the pack contract digest covers them. - Sidecar (`turn-dispatch-journal.ts`, `codex-runtime-adapter.ts`, `runtime-host.ts`, `acpx-runtime-sidecar.ts`): when the runtime port starts a turn, before any provider work, it truncates the journal and writes `received`. ACPX calls `onAcpMessage("outbound")` synchronously before it writes a message to the provider. On the turn's first `session/prompt` request the port appends `dispatched` and fsyncs it there, so the record exists before the bytes leave. A failed dispatch write aborts the prompt write. A failed `received` write removes the file and runs the turn unjournaled. - Runner (`acpx_turn_dispatch.rs`, `acpx_provider_backend.rs`): the journal is read only after the provider lifetime fence proves that the sidecar and provider are gone. A lone complete `received` record for the turn proves the prompt never left. - A start that failed in flight saves the session `prepared` with no active turn and returns `ACPX turn start failed before its prompt reached the provider`. The run's next `turn.start` reopens the conversation and sends the prompt once. Until the proof settles, the saved state stays `turn_starting`, so a Runner killed during the proof goes through the restore check below. - A Runner killed while the session was `turn_starting` restores the session idle (`session.resumed`, no `turn.failed`). The same-run resume then starts the turn once instead of sending a continuation that would lose the prompt. - A `dispatched` record, another turn's record, a torn or malformed file, or no file keeps today's behavior (closed session, or lost-turn recovery). - Fake sidecar: writes the same journal and adds `hang-before-dispatch` / `hang-after-dispatch` faults. - Docs: `packages/paperclip-runner/docs/durable-recovery.md` ("ACPX turn starts cut off before the prompt left"). The design note is `doc/plans/2026-11-10-durable-turn-start-and-tool-call-records.md` (in #20674). ## Verification - Red on master, green here: `cargo test -p paperclip-runner-core --test acpx_park_preparation cut_turn_start` (4 tests: cut before dispatch reopens and sends once; cut after dispatch stays closed and never resends; Runner killed before dispatch restores idle; Runner killed after dispatch or with no journal reports the lost turn). The two "before" cases fail on master. - `cargo test -p paperclip-runner-core`: all pass. `cargo fmt --check` clean. - `pnpm --filter @paperclipai/paperclip-runner exec vitest run src/drivers/acpx/turn-dispatch-journal.test.ts src/drivers/acpx/codex-runtime-adapter.test.ts`: 121 passed. One test uses a real ACPX runtime and an ACP peer that refuses a prompt arriving before its `dispatched` record; it fails if the hook is removed. - `node scripts/generate-acpx-sidecar-contract.mjs --check`, `node --test scripts/contracts/contract-fingerprints.test.mjs`, and `tsc -p tsconfig.test.json` pass. - Time to first token, measured with the same open/write/fsync sequence on this Mac (APFS, load average ~11, 300 samples): `received` p50 0.24 ms, p95 0.59 ms; `dispatched` (with fsync) p50 4.7 ms, p95 11.5 ms. Against a Claude TTFT budget of 9.5–16.5 s p50 this is under 0.1%. No wait, round trip or file transfer is added. Recovery work (fence, journal read) runs only after a failed start. - Live restart-chaos wave and runner image: see the comments on this pull request. ## Risks - A false "not dispatched" proof would duplicate a prompt. It needs a lone, complete `received` record for this exact turn ID, read after the fence proves the sidecar is gone. `dispatched` is fsynced before the prompt write, and a failed write aborts the send. A failed `received` write removes the file, so it cannot leave a lone record. - Upgrade safety: saved provider state gains no field (its decoder denies unknown fields), and only lifecycles an older Runner already handles are written. An older Runner ignores the journal file. State saved by an older Runner has no journal, which reads as unknown and keeps today's behavior. A new Runner with an older sidecar never sees a `received` record, so no proof forms. - The changed sidecar schema changes the pack contract digest, so Daytona needs a new runner image. Older image packs are not reused. - A failed start now waits for the provider lifetime fence (at most 5 s, usually immediate) before it settles. This is only on a failure path. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell, file editing) in Claude Code. Extended reasoning was used. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
|
52160eba35 |
test(restart-chaos): K15 redeploy, F1 workspace symlink, and the staging failure census (#20679)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - The restart-chaos harness (`tests/restart-chaos/`) restarts the
control plane while real agent tasks run. It grades whether every task
finishes without a human Retry.
> - The hosted staging stack deploys about every 40 minutes. In 48
hours, 276 runs failed and 63 tasks stayed blocked. About a quarter of
the failures came within 10 minutes of a deploy. The harness passed the
same areas.
> - A hosted deploy replaces the server container. The new server runs
in another process space and has a new deployment id. The PID-based "the
earlier owner exited" proofs never apply across that boundary. A harness
restart on one machine keeps the process space, so every PID proof
succeeds and the hosted recovery path is never tested.
> - This pull request adds K15, a redeploy fault. It also adds the
staging census that shows the gap.
> - The benefit: chaos waves can now exercise the recovery path that
staging runs after every deploy, on local and Daytona.
## Linked Issues or Issue Description
**What happened?**
Many staging tasks that a deploy crossed ended blocked. Examples are
`native_remote_recovery_lease_mismatch`,
`runner_remote_process_identity_unavailable`,
`prior_owner_terminal_state_indeterminate` and `PRP provider resume
state is unavailable`. The restart-chaos Daytona waves passed for the
same adapters.
**What did you expect?**
The chaos harness should reproduce the deploy conditions that staging
has, so that these failure classes show up in a wave first.
**Steps to reproduce**
1. Compare the staging run records' `controllerProcess.hostHash` and
`deployment.deploymentId` across a deploy. Both change on every deploy.
2. Compare the same fields across a harness K2 restart. Both stay the
same.
3. Read `controllerProcessExited` and `sameProcessSpace` in
`server/src/services/controller-process-identity.ts`. A different
`hostHash` means that no PID proof applies.
**Additional context**
The full census, the failure classes and the per-class harness gaps are
in `doc/plans/2026-11-11-staging-failures-vs-chaos-coverage.md`. The
task identifiers in it are neutral labels.
## What Changed
- New fault `K15` (redeploy), in `tests/restart-chaos/redeploy.ts` and
`redeploy-preload.mjs`. A K15 wave gives every server an emulated host
name through a `--import` preload, which changes the process-space hash.
It also gives Railway-shaped deployment variables with the instance home
as the exclusive volume. A K15 stops the server with SIGTERM and starts
the next generation, with a new host name and deployment id.
- `supervisor.ts`, `main.ts`, `config.ts` and `types.ts`: select K15
with `--faults K15`. It works on `--env local` and `--env daytona`. The
receipt records `redeploy` and the timeline records `redeploy_switched`.
Config refuses K15 together with MS1–MS3.
- New opt-in workload `F1` (`workloads.ts`): the agent keeps an absolute
symlink in its repository workspace. A staging save-back refused such a
link and blocked the task. The oracle needs `done` and the saved notes
file, with no Retry. `f1-symlink.test.ts` covers the prompt, the grading
and the option planning.
- `redeploy.test.ts`: identity derivation, the environment, the argument
order, a real child process that proves the preload changes both CJS and
ESM `hostname()`, option parsing and plan detection.
- `tests/restart-chaos/README.md`: a K15 section and a row in the
"Pieces" table.
- `doc/plans/2026-11-11-staging-failures-vs-chaos-coverage.md`: the
48-hour staging census, 12 failure classes, the deploy timeline, the
reason the harness missed each class, and proposed scenarios.
## Verification
- `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts
tests/restart-chaos/redeploy.test.ts
tests/restart-chaos/scheduler.test.ts
tests/restart-chaos/repeated-restarts.test.ts`: 68 tests pass. The full
restart-chaos unit suite passes: 43 files, 516 tests.
- `node scripts/typecheck-test-projects.mjs tests/restart-chaos`:
passes.
- Live Daytona waves on
|
||
|
|
e33ede9a8a |
fix(native): settle a restart-orphaned write_document from its transactional receipt (#20674)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native run calls Paperclip tools such as `write_document`. The runner saves each tool input durably before the server runs the tool, and the server saves a result receipt > - A control-plane restart can cut a `write_document` call after the runner saved its input and before any result was saved > - Recovery then has no proof of the outcome. A call from an earlier server process settles as "unknown outcome" (#20628). A call the current server process admitted stays pending, close cannot prove that the provider drained, and the session quarantines (`native_session_cleanup_quarantined`). A person must click Retry > - `write_document` saves its revision and its idempotency receipt in one database transaction, under the run and issue row locks. The revision is also compare-and-swap on `baseRevisionId`. So the receipt is a durable outcome record: present means applied, absent means not applied > - This pull request makes recovery use that proof. Recovery runs the same receipt-guarded write. It replays the saved result, or it applies the write exactly once > - The benefit is that an orphaned `write_document` settles with its real result. The document is never written twice, and the run does not need Retry for this call ## Linked Issues or Issue Description **What happened?** Daytona restart-chaos wave `w-20261110T174930-s9205`, task t01 (codex, workload C1, fault RR2). A restart cut a `write_document` call. The close settlement showed `semanticTools.pending: [{ operationId: "write_document", callId: "exec-8b79ab78-…" }]` and `providerDrained: false`. The session quarantined with `native_session_cleanup_quarantined`, and the task needed Retry. Since #20628, a call that an earlier server process admitted settles as `tool_call_interrupted_by_restart` with an unknown outcome. The agent then has to guess whether the write applied. **Expected behavior** When a durable record proves the outcome, recovery uses it. A write that applied returns its saved result. A write that did not apply runs once. When the record cannot prove it, the call stays as it is today. **Steps to reproduce** 1. On master, run `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/paperclip-runner-tool-authority.test.ts -t "orphaned"`. 2. `authority.recover()` returns `null` for a `write_document` that has no receipt, so the call stays unsettled. Refs #20628, #20639, #20507 ## What Changed - `PaperclipRunnerToolAuthority.recover()` now settles `write_document` through its transactional receipt (`#recoverTransactionalWrite`). It runs the same receipt-guarded mutation as an ordinary call. If the receipt exists, it returns the saved result and writes nothing. If no receipt exists, it applies the write once. - A definite refusal that rolls the transaction back (a 4xx such as a stale `baseRevisionId`, a reused idempotency key, invalid input) settles as a tool error. Any other failure leaves the call unsettled, as before. - The set of operations with this proof is explicit: `TRANSACTIONAL_RECEIPT_OPERATIONS` contains only `write_document`. Other tools keep their current recovery rules. - New test: an orphaned `write_document` with no receipt applies once; a second recovery and a late original call replay the receipt; an applied write recovers its saved result; a stale base settles as a tool error with no change. - Updated test: a reused idempotency key with other input now settles as an `paperclip_runner_tool_idempotency_conflict` tool error instead of staying pending. The receipt proves that this call can never apply under that key. - Docs: `packages/paperclip-runner/docs/durable-recovery.md` (semantic operations that outlive a turn) and a new design note, `doc/plans/2026-11-10-durable-turn-start-and-tool-call-records.md`. The note also covers the second change (an ACPX turn start cut off before the prompt reached the provider), which comes in a separate pull request. ## Verification - Red on master, green here: `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/paperclip-runner-tool-authority.test.ts` (43 passed). - `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`: 178 passed. One unrelated ACPX timing test failed once under load and passed when run alone. - Server typecheck: `npx tsc --noEmit -p server/tsconfig.json` passes. - `pnpm verify` exits 2 (a runner doc changed), so required CI supplies the broad coverage. - Live restart-chaos wave: see the comment on this pull request. ## Risks - Recovery can now apply a write that never applied before the restart. This is the call the agent made, applied once. The write is compare-and-swap on `baseRevisionId`, so it cannot overwrite a revision that someone wrote after the agent read the document. That case settles as a conflict error. - Upgrade safety: no saved shape changes. Receipts saved by the previous release have the same shape. A server on the previous release keeps its current recovery for these calls. - Time to first token: no change. This code runs only during recovery. - No runner source changes, so no new runner image is needed. ## Model Used Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell, file editing) in Claude Code. Extended reasoning was used. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
|
9296c74e8d |
fix(native): let a restart claim take back its lapsed lease at budget admission (#20650)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - After a control-plane restart, restart recovery claims an interrupted native run (`resume_dead_runner`) and grants a 20-minute lease. Budget admission then re-checks that the claim still owns the run before the resumed run may start provider work. > - A host sleep, or the restart-chaos fault TM2, can make that stored lease look expired while its owner is alive. #20625 lets the owner take back such a lease when the row still shows exactly its own claim. That rule did not cover budget admission. > - So a claim whose lease lapsed before dispatch was refused with `Native recovery no longer owns provider dispatch`. The run failed with `adapter_failed`, recovery held it with `native_continuation_requires_reconciliation`, and the user had to click Retry. > - This pull request applies the #20625 rule in budget admission: the claim takes its lease back through the existing re-acquire condition. > - The benefit is that a run resumed after a restart survives a time jump before dispatch with no Retry. Every fence and every budget check stays the same. ## Linked Issues or Issue Description Refs #20625, #20474 **What happened?** Restart-chaos wave L7 (seed 9107, local) on a branch with #20625, task codex B5 TM2. A TM3 restart claimed the run (`resume_dead_runner`, generation 2). TM2 then aged every database time by one hour before the resume reached dispatch. Budget admission (`lockRunBudgetRecoveryOwnerInTransaction`) required a live lease and threw `Native recovery no longer owns provider dispatch`. The run failed with `adapter_failed`, recovery stopped with `native_continuation_requires_reconciliation` / `preserve_without_replay_v1`, and the task needed Retry. **Expected behavior** The claim's own lapsed lease is taken back, and the resume is admitted when the budget allows it. **Steps to reproduce** `node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env local --adapters codex,claude --workloads A1,B5,C1 --faults TM1,TM2,TM3 --time-age-hours 1 --concurrency 3 --tasks 6 --seed 9107` (timing-dependent). ## What Changed - `native-execution-lease.ts`: two new helpers. Both use the #20625 own-claim rule. Neither takes a lock out of order. - `readOwnNativeRecoveryClaimLease` (read-only) returns `live`, `lapsed_own_claim` or `lost`. It compares the lease against the database clock. A lapsed lease counts as `lapsed_own_claim` only when all of these hold: - the same lease owner; - this process's controller boot, pid and process start; - a non-null `recovery_state`; - `nativeExecutionLeaseHeldCondition({ remote: false, controllerGeneration })`, which applies the #20625 own-claim rule and every #20474 untouched check; - no non-local environment lease on the run. - `extendOwnNativeRecoveryClaimInTransaction` extends such a lease by the local execution TTL, with one conditional UPDATE on the same predicate. - `budget-reservations.ts`: - `lockRunBudgetRecoveryOwnerInTransaction` still refuses another owner, and refuses a lapsed lease unless it is the run's own untouched claim. It no longer locks the run row, so the documented coordinator → issue → run order holds. - The lease is extended only at the end of an admitted reservation (`reserveRunBudget`) or an admitted dispatch recheck (`recheckRunBudgetReservationInTransaction`). At those points the caller already holds the run row lock. A refused admission, for example one over budget, leaves the lease lapsed. - `heartbeat-native-restart-suspension.test.ts`: the test "still settles a recovery whose claim merely expired" encoded "expired means not the owner". #20474 and #20625 changed that rule for an untouched own claim. The test now has two cases: - an expired claim that another actor recorded on (`awaiting_evidence`) still settles with no provider session; - an untouched expired claim is dispatched. - `doc/plans/2026-11-09-explicit-native-lease-claim-tokens.md`: Step 1 lists the new user of the rule. ## Verification - New `server/src/__tests__/native-recovery-dispatch-lease-reacquire.test.ts`: - `reserveRunBudget` takes back the claim's own lapsed lease and admits the resume. This fails before the change. - An over-budget resume is still refused, and its lease stays lapsed. - Fenced variants each throw `Native recovery no longer owns provider dispatch` and leave the row unchanged: a graceful-shutdown suspension, an appended failure settlement, `awaiting_evidence`, `blocked`, a claim from an earlier generation, another boot, a remote sandbox lease, and another lease owner. - At the dispatch gate (`recheckRunBudgetReservationInTransaction`), an admitted own claim gets its lease extended, and a fenced claim is refused with the row unchanged. - Related suites pass (308 tests): `native-execution-lease-reacquire.test.ts`, `closed-native-receipt-accounting.test.ts`, `heartbeat-native-predispatch-retry.test.ts`, `run-dispatch/adapters/postgres.test.ts`, `heartbeat-native-restart-suspension.test.ts` and `heartbeat-restart-claim-lapse.test.ts`. - `cd server && npx tsc --noEmit` passes. - Live: one L7 wave (seed 9107) ran on the first commit of this branch (`w-20261110T212307-s9107`). It logged 0 `Native recovery no longer owns provider dispatch` refusals, against 1 in the previous wave. Codex B5 TM2 got past dispatch, but it still needed Retry for another reason: after a `runner.drain` timeout, the scheduled same-run resume did not send the continuation of #20645. That case is tracked separately. The other five tasks passed. ## Risks - This widens when a restart claim can take back its own lapsed lease, and only at budget admission. It is local only. It needs the same owner, boot and generation, and the row must show exactly that claim. A claim that any other actor touched still fails as before. - Budget hard-stop behaviour is unchanged. The lease is extended only after budget admission passes. - No schema change and no migration. Rows saved by earlier releases carry the same history shape. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test runs). Extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
|
30403781b9 |
fix(native): keep crediting proven unstarted restarts past the generation limit (#20647)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native run survives a control-plane restart through restart recovery, which resumes the interrupted provider turn > - Restart recovery charges a provider attempt only when the provider did work that could be lost, and it stops after three charged attempts > - From controller generation 12 on, every restart charged an attempt before that work rule was checked > - In a busy restart wave, one long task sees 12 to 15 restarts, most caused by other tasks. A restart that hit before the provider accepted a turn was charged, the budget ran out, and a person had to click Retry > - This pull request keeps the work rule past generation 12 for restarts that a durable record proves did not start the provider, and bounds a crash loop with a count of consecutive restarts without provider progress > - The benefit is that a long task survives many unrelated restarts without a manual Retry, while a run that never makes progress still stops ## Linked Issues or Issue Description Refs #20625 (same restart recovery path). No public issue exists for this case. **What happened?** `restartResumeKeepsAttempt` returned "charge" for every restart once the run reached `NATIVE_PROGRESS_CREDIT_GENERATION_LIMIT` (12) controller generations. It did this before it read the work-based outcome. In a local restart chaos wave (L6, seed 9106, task t03, Claude B5 with RR1) the run reached generation 13. The last restart interrupted a resume whose runner started but whose provider had not accepted a turn. That restart was charged at the last attempt, so the claim failed with `execution_recovery_budget_exhausted` and `native_restart_recovery_blocked`. The task needed a manual Retry. **Expected behavior** A restart that a durable record proves did not start provider work does not spend an attempt, also past generation 12. A crash loop that never makes provider progress still stops automatically. When it stops, the task shows a Retry that works. **Steps to reproduce** Run a native task past controller generation 12. Make the last restart hit after `native.process_start_requested` but before `turn.accepted`, with the provider attempt at 3. The new integration test `does not charge a restart past the generation limit whose resume never started the provider` reproduces the failure before the fix. **Paperclip version or commit** `master` at |
||
|
|
ed5e7493bb |
fix(native): take back a lapsed lease held under the run's own restart claim (#20625)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native run holds an execution lease on its `native_run_finalizations` row. The owner renews it every 5 minutes. After a host sleep (or the restart-chaos time fault TM2), the stored lease looks lapsed although the owner is alive. > - #20474 lets the same owner take back a lapsed lease when no other actor recorded anything on the row. One of the checks is "`recovery_state` is null". > - Restart recovery writes `recovery_state` (`resuming_session`, `awaiting_runner_reattach` or `bootstrap_incomplete`) in the same update that grants the lease. The executor keeps that value until a result is accepted. > - So a run that a restart resumed can never take back its own lapsed lease. Renewal interrupts the turn, and failure settlement is refused with `native_session_lease_lost`. The run fails with `adapter_failed`, recovery holds it with `preserve_without_replay_v1`, and the user must click Retry. > - This pull request counts the claim's own record as untouched when the row still shows exactly that claim and nothing recorded since. > - The benefit is that a resumed run survives a later host sleep with no Retry. Every real takeover still fences the late owner out. ## Linked Issues or Issue Description Refs #20474 **What happened?** The zero-retry restart-chaos matrix (local wave L7, seed 9107: TM1, TM2, TM3 with 1 h aging) had 3 of 6 tasks that needed Retry. All three were Codex runs. In each run, a TM3 kill made restart recovery resume the run (`resume_dead_runner`, generation 2). The runner restored the conversation, reported `provider_turn_lost_on_restore`, and the same-conversation restart continuation started as designed. Then the TM2 fault aged every database time by one hour. The execution lease lapsed while its owner was alive. The run then failed with `native_session_lease_lost` (`adapter_failed`), and recovery stopped with `native_continuation_requires_reconciliation` / `preserve_without_replay_v1`. One run had already proposed its result and completed its turn. The Retry click worked every time. **Expected behavior** A run whose lease lapsed only because time jumped, and that no other actor claimed, continues with no Retry. This must also be true after a restart resumed that run. **Steps to reproduce** `node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env local --adapters codex,claude --workloads A1,B5,C1 --faults TM1,TM2,TM3 --time-age-hours 1 --concurrency 3 --tasks 6 --seed 9107` ## What Changed - `server/src/services/native-runtime/native-execution-lease.ts`: the "untouched" part of `nativeExecutionLeaseHeldCondition` now also accepts the run's own local restart-recovery claim. All of these must be true: - the last `recovery_history` entry is that claim: same controller boot and generation as the row; - its disposition matches the current `recovery_state` (`resume_dead_runner` → `resuming_session`, `reattach_existing_runner` → `awaiting_runner_reattach`, `bootstrap_incomplete` → `bootstrap_incomplete`); - `recovery_request_id` equals the request that claim recorded. Any later actor appends a history entry (shutdown suspension, failure settlement), moves the state (`awaiting_evidence`, `blocked`), or replaces the owner, boot or generation. Each of these still fences the late owner out. All other untouched checks are unchanged. - `native-execution-lease-reacquire.test.ts`: new cases for each claim kind, a requested claim, failure settlement under the claim, and eight fenced variants. - `doc/plans/2026-11-09-explicit-native-lease-claim-tokens.md`: Step 1 lists the new rule. ## Verification - Red/green: `cd server && npx vitest run src/services/native-runtime/native-execution-lease-reacquire.test.ts`. Without the source change, 5 new cases fail (the three claim kinds, the requested claim, and failure settlement). With it, all 37 pass. - Related suites pass: `native-execution-lease.test.ts`, `heartbeat-restart-claim-lapse.test.ts`, `heartbeat-native-orphan-sweep-lease-race.test.ts`, `native-runner-restart-lease-renewal.integration.test.ts`, `native-runner-restart-preparation.integration.test.ts`. - `cd server && npx tsc --noEmit` passes. - Live: the same L7 wave (seed 9107, local) on this branch. The server log has 0 `native_session_lease_lost` (the baseline had 3). The two tasks whose cells failed this way in the baseline (codex C1 TM1, codex C1 TM3) now pass with no Retry. Two other tasks needed Retry for different causes, which this pull request does not change: codex B5 TM2 (a `runner.drain` timeout, then a same-run retry that may not continue a turn lost on restore) and claude C1 TM2 (`ACPX session is not open` on `turn.start` until the retry budget ran out). The L6 wave (seed 9106) also ran on this branch: 5 of 6 passed. Its one failure is `execution_recovery_budget_exhausted` and is not related to leases. ## Risks - The rule widens when an owner may take back its own lapsed lease. It applies only to local runs, before a result, with the same owner token, attempt, boot, pid, process start and generation. A claim that any other actor changed afterwards does not match. - Rows saved by earlier releases carry the same history entry shape, so a run claimed before a deploy is handled the same way. A deploy changes the boot, so the new process never takes back the old process's lease. - No schema change and no migration. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test runs). Extended reasoning. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
|
1b7ec13acb |
perf(native): default remote sandbox runs to a warm, reusable sandbox for measured harnesses (#20048)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents can run in remote sandboxes (Daytona), where each turn needs
the task's workspace and a runner inside the sandbox
> - With a default environment, every remote turn got a brand-new
sandbox: the whole workspace was uploaded again and a new runner
started, even for a follow-up comment
> - In a paperclip-sized workspace that made a follow-up take 30 s to
first token, against 8–10 s with a reusable warm sandbox
> - This pull request makes an agent that never chose a lifecycle run
warm on a sandbox for the measured harnesses (Codex, OpenCode, Claude),
so a follow-up keeps its sandbox, runner and workspace
> - The default applies only where warm can work: a sandbox provider
that keeps reusable leases, and a managed AI account that supports a
warm session. Everywhere else the run is per-turn, exactly as before
> - The benefit is fast remote follow-ups by default, without
re-uploading a large repository every turn
## Linked Issues or Issue Description
**What happened?**
On Daytona, with a fresh paperclip clone as the project workspace, a
follow-up comment took 30 s to its first token. The run log showed
`environment: fresh/created` on every follow-up. It spent 13.3 s in
`workspace_sync.transfer`, a full upload into the new sandbox, and 8–9 s
in `runner.session.resume`.
**Steps to reproduce**
1. Create a Daytona environment with the defaults: lifecycle inherited,
lease reuse off.
2. Run a Codex task in a large repository, then post a follow-up.
3. The follow-up gets a new sandbox, uploads the whole workspace again
and starts a new runner.
**Expected behavior**
A follow-up on the same task reuses the live sandbox, its runner and its
workspace (plan rule 2: no full re-upload per turn).
## What Changed
- **One rule for the default.** `defaultsRemoteSandboxRunToWarm`
(`native-runtime/native-warm-capability.ts`) decides the remote-sandbox
warm default for both the lease (environment view) and the runner
(native session policy), in one vocabulary (runtime backend plus ACPX
agent). It is applied only when no lifecycle was chosen anywhere:
admitted session, environment `runnerLifecycleMode`, or agent
`lifecycleMode`. It holds only when:
1. the harness is measured (Codex, OpenCode, ACPX Claude);
2. the sandbox provider keeps reusable leases (Daytona, CreateOS and
boxd declare them; E2B, Modal and the built-in `fake` do not); and
3. with a managed AI account, that harness supports a managed warm
session (`supportsManagedNativeWarmSession`: Codex).
- **The harness resolves as the runtime resolves it.**
`resolveWarmProviderSelection` uses
`resolvePaperclipRunnerProviderProfile`, so an omitted `provider` is
Codex and ACPX without an agent is Claude, in both decisions. A config
the runtime would refuse gets no default.
- **Lease side** (`runner-environment-lifecycle.ts`,
`environment-run-orchestrator.ts`): for an eligible run only, the
orchestrator asks the provider before acquisition
(`providerSupportsReusableLeases` in `environment-runtime.ts`: the
acquisition rule, one registry lookup for a plugin provider, no
worker-ready wait). With support, the run-scoped view sets `reuseLease:
true` and forwards the warm idle window to the provider's auto-stop. The
shared environment is never rewritten. Every other run resolves exactly
as on master, without the probe.
- **Runner side** (`native-lifecycle.ts`, `runtime-selection.ts`): the
native policy applies the same rule to the acquired lease
(`reusableLeaseConfigured && effectiveCapabilities.reusableLeases`). If
acquisition downgrades the lease anyway, the run falls back to per-turn;
a default never produces
`runner_warm_environment_requires_reusable_lease`.
- **Idle window.** The remote default uses the window the environment
view forwards to the provider: an inheriting environment's
`runnerIdleTimeoutMs`, then the agent's `idleTimeoutMs`, then 5 minutes.
The sandbox execution target now carries the environment's window
(`runnerIdleTimeoutMs`, an optional field in `adapter-utils`'s execution
target) so the runner and the provider auto-stop agree. An explicit warm
choice and the local default are unchanged.
- Unchanged: explicit agent choices, environments that pin a lifecycle,
plugin-task environments, admitted sessions (`persistedPolicy`), and
local behavior.
- `tests/restart-chaos`: the vCPU budget check counts `--daytona-inherit
--lifecycle default` sandboxes as retained, because they now default to
warm.
- `doc/plans/2026-11-08-large-workspace-speed.md`: the measurements, the
ranked waits, and the gate.
## Verification
- Unit and integration tests, one per rule:
- Rule 1 (provider gate): `native-lifecycle.test.ts` (ephemeral lease →
per-turn), `runner-environment-lifecycle.test.ts` (no reusable leases →
environment unchanged, no warm window),
`environment-runtime-reusable-lease-probe.test.ts` (`fake`, an
undeclared plugin, a declared plugin whose worker does not verify the
reuse methods, and a plugin without a running worker read false),
`runtime-selection.test.ts` ("defaults an agent without a lifecycle on
...": reusable → warm/keep_running; reuse configured on an unsupported
provider → per_turn/destroy_after_turn; ephemeral → per_turn),
`environment-run-orchestrator.test.ts` (probe result decides
`reuseLease` and the forwarded window).
- Rule 2 (explicit configuration wins): explicit per-turn agent, pinned
environment, admitted session, configured `idleTimeoutMs`, managed
Claude per-turn, managed Codex warm; the orchestrator does not probe
when a lifecycle is explicit.
- Rule 3 (omitted provider): `{}` resolves to Codex in the environment
view; `runtime-selection.test.ts` "agrees with the environment view's
lease for ..." feeds the environment view's lease into selection for
Codex, a provider-less Codex agent, OpenCode, ACPX Claude, ACPX without
an agent, and Copilot (per-turn), and checks both decisions agree. A
plugin-task target stays out of the default.
- The two suites that failed on the previous head
(`runtime-selection.test.ts`,
`workspace-recovery-preparation-dispatch.test.ts`) pass locally.
Locally, 6 connector-skill cases in `runtime-selection.test.ts` fail
with `ENOENT .claude/skills/slack/SKILL.md` because this checkout is a
nested worktree; they do not touch this change and pass in CI.
- End-to-end check of this default on Daytona (before the gate was
added; Daytona keeps reusable leases, so the gate does not change this
path): a user-default environment (lifecycle inherited, lease reuse
unset, `--daytona-inherit`), a fresh paperclip clone, built from the
image's source revision with this change (branch measure/remote-warm).
| | Before (per-turn sandbox) | After (this change) |
|---|---|---|
| Follow-up, Codex | 30.4 s | 7.6 s |
| Follow-up, Claude | about 30 s | 11.0 s |
| First follow-up after an idle K1, Codex | 45.9 s | 20.3 s |
| First follow-up after an idle K1, Claude | 38.3 s | 18.3 s |
The run-log reuse records show the follow-ups reusing the workspace,
environment, runner and provider session, with no
`workspace_sync.transfer`. First turns still upload the workspace once
(14.8 s and 26.1 s). Every sandbox was cleaned up.
- Daytona restart chaos wave (K1/K2/K4 with a mid-run comment,
inheriting environment, agents with no lifecycle, runner image built
from this branch): wave `w-20261110T171438-s20048`, seed 20048, on
|
||
|
|
bf0220d34a |
perf(workspaces): migrate with one archive and no readback (#20554)
## Thinking Path > - Paperclip coordinates agents and keeps their workspace files. > - A legacy workspace must move from a local volume to object storage. > - The importer creates an archive, downloads and restores it, and captures the source again. > - Parent imports also download and recapture children that already have committed snapshots. > - This repeated work makes the one-time migration slow. > - This change copies each source file once and trusts the upload acknowledgement. > - Existing source locks and atomic publication still own the migration. ## Linked Issues or Issue Description **What existing behavior does this improve?** The one-time migration of legacy local and Git workspaces to durable snapshots. **Current behavior** Each workspace is captured twice and restored as a test during migration. Parent imports also read back already imported child snapshots. Capture hashes source files before writing them and scans them again afterward. **Proposed behavior** Build one archive from one source-file read, encrypt it, upload it, and publish its reference. Trust the upload acknowledgement. Reuse journaled uploads after an interrupted process. Retain committed child snapshots by database reference. **Reason and benefit** Remove repeated source reads, temporary copies, complete downloads, and recursive child verification. The tests assert one capture, one upload, and zero migration downloads. **Breaking changes** The archive format stays compatible. File hashes required by current readers are computed inline during the copy. There is no preliminary file-hashing pass. Existing readers can cold-restore the result. Related work: #19817 and #20447. ## What Changed - Add single-pass capture that fills fixed-length checksum fields in the existing archive header after copying. - Remove migration readback, test extraction, fresh-source recapture, and Git integrity scans. - Compute the encrypted archive checksum while writing it. - Reuse acknowledged and journaled uploads without rebuilding or downloading them. - Retain already imported children through their database bindings. Remove the recursive payload verification cache. - Keep company checks, source and writer locks, recovery bindings, journal identities, cancellation, and atomic publication. - Bind archived repository intent paths to the operation without reconstructing an archive to check them. - Update the migration contract and the affected tests. ## Verification - Node 24.22.2 and pnpm 9.15.4 on macOS arm64. - At |
||
|
|
f4337bb5fb |
test(evals): land the chat catalog eval harness on master (#19614)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The chat-feature evals measure whether agents complete real chat requests through the product > - The chat catalog eval harness and its repairs live on the `chat-eval/master-candidate` branch, not on master > - Today's work made 139 of 242 chat evals pass in verified single-case live runs. Each product fix landed on master as its own pull request > - The eval-side changes (setups, graders, fault seams, evidence capture) still exist only on the candidate > - This pull request lands the harness on master. Master stays the source of truth for product behavior > - The benefit is one tree: evals run against master commits, and product fixes reach the evals as soon as they merge ## Linked Issues or Issue Description Refs #19441, Refs #19599, Refs #20270, Refs #20334, Refs #20342, Refs #20344, Refs #20354, Refs #20360, Refs #20373, Refs #20382, Refs #20409 **What existing behavior does this improve?** The chat catalog eval harness and the product it evaluates live on a long-running branch outside master. Eval fixes do not reach master, so master-based eval runs use old setups and graders. **Subsystem affected** `tests/runner-e2e` chat catalog harness, plus a small set of product changes that exist only on the candidate. **Proposed change** Land the harness on master. Every product fix from this eval work already merged to master as its own pull request. Conflicts keep master's product behavior. ## What Changed - Most of the diff (808 of 851 files) is under `tests/runner-e2e/`: chat-catalog domains, scenarios, descriptors, the TestOps manifest, private evidence tooling, and their tests. - Graders now check saved state, setup literals and rendered or clicked links. Many keyword and regex checks on free-form replies are removed. Each changed grader has a bumped `oracleVersion` and calibration tests (a valid reply, a reworded valid reply, and broken variants that must fail). - The shared link helper `hasLink` no longer treats a list link (for example `/COM/issues?view=mine`) as a link to every task without a URL key. - Setup fixes build real state through supported product routes: routine history, agent grants through the board grant route, budget holds with a real cap, and a storage-layer archive fault for `inbox-search.archive-partial-failure`. - Browser boundary journeys (sign-in, chat-disabled, onboarding) now write the standard pass evidence files. - Retargeted cases, approved by the maintainer: `agent-chat.chat-disabled` (Conference Room gate), `agents.restore-instructions` (no saved instruction history), `access.manage-membership` (member permission edits stay board-only). - The private evidence trusted files (`manual-private-handoff.ts`, `manual-private-retention.ts`, `manual-private-report-assets.ts`) are byte-identical to master. - Product files outside `tests/` (43 files, +1,328 −49) carry candidate-only changes that master does not yet have. Most are tests. The main code changes are in `request-human-input.ts`, `heartbeat-run-summary.ts`, `native-completion-feedback.ts`, `native-run-finalizer.ts`, `openapi.ts`, `secrets.ts` and `IssueChatLongThreadPerf.tsx`. ## Verification - `pnpm vitest run chat-catalog-domains chat-catalog-contract.test.ts chat-catalog-consistency.test.ts unit-launcher.test.mjs --testTimeout=180000` in `tests/runner-e2e`: 269 files pass, 3,550 tests pass. - `node --test chat-full-catalog-testops-manifest.test.mjs`: 10 pass. - `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`: no change. - `tsc` for `server` and `tests/runner-e2e`: clean. - Live single-case verification: 139 of 242 cases passed on paid runs, and a person read the saved evidence for each. These are single-case runs, not one full-cohort run. ## Risks - Large diff. Most of it is test-only, but the 43 non-test files change product code paths. Review those files first. - Some graders are now less strict on reply wording. A person must read replies for meaning on those cases. - 42 cases need disposable outside accounts (Notion, Slack, GitHub, Daytona and others) and stay blocked. `access.invite-teammate` stays failing until local agent isolation lands, because the runner's durable state keeps a live invite token. - The paid workflow compares the trusted private-evidence files with master. They are unchanged, so this risk is low. ## Model Used Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution, in Claude Code. Subagents used the same model. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
dc9971bdb8 |
fix(native): keep another server's orphan sweep off a lease its owner took back (#20508)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native run holds an executor lease on its `native_run_finalizations` row. #20474 lets the same owner take back its own lease when the lease lapsed only because the host slept. > - That take-back is safe only if every other actor that acts on a lapsed lease records its action under the coordinator row lock first. The plan `doc/plans/2026-11-09-explicit-native-lease-claim-tokens.md` lists the actors that do not. Gap 2 is another server's orphan sweep. > - The sweep reads the lease as lapsed. Later it writes the ownership-unverified marker on the run, without the coordinator lock and without a new check of the lease. If the owner wakes and takes its lease back between the read and the write, a healthy run gets the marker. > - The board then shows "automatic recovery is blocked" on a run that works. After the next host sleep, the owner cannot take its lease back, so the turn is interrupted and the user must click Retry. The owner's rule is that a task interrupted by sleep must resume by itself. > - This pull request makes the sweep write its marker under the coordinator row lock, and only while no executor lease on the run is live. > - The benefit is that exactly one side wins: the sweep marks a run whose owner did not come back, and it leaves alone a run whose owner took its lease back. ## Linked Issues or Issue Description Refs #20474 **What happened?** Two servers share one database. Server A's host sleeps, so the executor lease of its native run lapses. Server B's orphan sweep reads the lapsed lease. Server A wakes and takes its lease back (#20474). Server B then writes `native_execution_ownership_unverified` on the run that A still owns. **What did you expect?** Server B writes the marker only if the lease is still lapsed when it writes. One side wins. **Steps to reproduce** `server/src/__tests__/heartbeat-native-orphan-sweep-lease-race.test.ts` holds the owner's take-back open, runs the real `heartbeatService(db).reapOrphanedRuns()` until its write waits on a lock, then commits the take-back. Before this change the run ends with `errorCode: native_execution_ownership_unverified` and the owner's next take-back fails with `native_session_lease_lost`. ## What Changed - `setRunStatusFromLive` (`server/src/services/heartbeat/run-lifecycle.ts`) has a new option `onlyWhileNativeLeaseLapsed`. It locks the coordinator row first. This is the same order that `lockNativeExecutionLeaseForReacquire` uses. The UPDATE then requires that no executor lease on the run is live. - `markNativeOwnershipUnverified` (`server/src/services/heartbeat/recovery.ts`) takes `leaseLapsed`. The orphan sweep in `recoverActiveExecutions` sets it when it read the lease as lapsed or unset. Other callers do not change. They keep the same call shape. - A new regression test, red before the fix: `server/src/__tests__/heartbeat-native-orphan-sweep-lease-race.test.ts`. It also checks that the sweep still marks a lapsed run whose owner did not come back, and that the late owner stays fenced out. - `doc/plans/2026-11-09-explicit-native-lease-claim-tokens.md`: gap 2 is marked closed without the epoch design. ## Verification - `cd server && pnpm exec vitest run src/__tests__/heartbeat-native-orphan-sweep-lease-race.test.ts` — red on `origin/master` (the run keeps the marker), green with this change. - `cd server && pnpm exec vitest run src/__tests__/heartbeat-native-orphan-sweep-lease-race.test.ts src/services/heartbeat/recovery.test.ts src/services/native-runtime/native-execution-lease-reacquire.test.ts src/__tests__/heartbeat-status-preserving-telemetry.test.ts` — 129 passed. - `cd server && pnpm exec tsc --noEmit -p tsconfig.json` — clean. - Restart-chaos time-fault wave on this branch: see the wave comment below. ## Risks - Low risk. Only the orphan sweep's marker write changes, and only for a run whose lease it read as lapsed. The sweep now takes the coordinator row lock before the run row. The owner's take-back uses the same order, so the two cannot deadlock. - If the owner took the lease back, the sweep now writes nothing and appends no event. The next sweep sees a live lease and defers, as before. - The other gaps in the plan (workspace finalization, shared-workspace holder detection, board copyback confirmation) do not change. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with tool use (shell, file edits, test runs). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude <noreply@anthropic.com> |
||
|
|
1402e92d1f |
fix(native): take back an executor lease that lapsed only because the host slept (#20474)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - A native run holds an execution lease on its `native_run_finalizations` row. The lease lasts 20 minutes, and the owner renews it every 5 minutes. > - When a laptop sleeps for longer than the lease, the wall clock and the database clock move past `lease_expires_at`. The server's monotonic timers barely move. > - After the wake, the next renewal finds its own lease expired. It interrupts a healthy turn with `native_session_lease_lost`, and the user must click Retry. Nobody else took the lease. > - Several background consumers act on an expired lease without changing its owner. So expiry alone cannot simply be ignored for the owner. > - This pull request makes takeover explicit. The same owner can take back its own lapsed lease only when no other actor has recorded anything on the row or the run. > - The benefit is that a run interrupted only by host sleep continues with no Retry. A real takeover still fences the late owner out. ## Linked Issues or Issue Description **What happened?** The restart-chaos time fault TM2 stops the server and runner tree and ages every database time by one hour, as a stand-in for a laptop that slept. After the thaw, the next 5-minute renewal of the native execution lease found the lease expired. It interrupted the turn (`turn.interrupted` at 300 s, `native_session_lease_lost`). The failure settlement was then refused with the same code, and the UI showed "Couldn't finish this attempt" with Retry. The owner, attempt, boot and pid were unchanged. Evidence: wave `w-20261110T051535-s304` t01 (TM2, 1 h) failed R3. t02 in the same wave (TM1, a 113 s server-only freeze with no clock jump) passed, so the failure follows the clock jump and not the pause. **Is this a harness artifact?** No. TM2 ages the stored rows, and a real sleep moves the clock forward. Both produce the same comparison: a stored `lease_expires_at` is earlier than `clock_timestamp()` after the wake. The server's renewal timer and local deadline use the monotonic clock. On macOS and Linux that clock does not count suspended time, so they do not notice the sleep in either case. The first database check after the wake is the renewal predicate `lease_expires_at > clock_timestamp()`, and it fails in both cases. **Expected behavior** A run whose lease lapsed only because time jumped, and that no other actor claimed, keeps running and finishes without Retry. **Steps to reproduce** `node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts --env local --adapters codex --workloads B5 --faults TM2 --time-age-hours 1 --concurrency 1 --tasks 2 --seed 304` with the harness from #20458. Refs #20458, #20500 ## What Changed - `native-execution-lease.ts`: add `nativeExecutionLeaseHeldCondition` and `lockNativeExecutionLeaseForReacquire`. A lease is held when it is live. For a local run, it is also held when it lapsed and is untouched. "Untouched" means all of these are true: - the claimed generation matches; - the phase is `observed` and no result is accepted; - `recovery_state`, `recovery_request_id`, `failure_code` and `next_attempt_at` are null; - `controller_host_hash` is not the board's attested-stopped stamp; - the run is still `running`, without the ownership-unverified error code and without a `nativeStartupCleanup` marker. - The caller's WHERE already matches the owner token, attempt and controller boot/pid/start. - The lock helper locks the coordinator row. If the lease lapsed, it also takes a share lock on the run row. The predicate then reads the latest committed run, and an in-flight run write cannot slip past it. - Renewal (`startNativeSessionExecutionLeaseRenewal` / `renewNativeSessionExecutionLease`) takes back a lapsed lease when the executor passes its claimed generation. Remote targets never do. A deadlock (40P01) on that rare path retries at the next tick, as a statement timeout already does. - `native-session-executor.ts`: the executor passes its generation to renewal. Failure settlement and the final release use the same lock and condition. After a result is accepted, a lapsed lease can only be relinquished (set to null), never renewed, because workspace finalization may act on it. - New `native-execution-lease-reacquire.test.ts` (embedded Postgres, 24 cases). - `doc/plans/2026-11-09-explicit-native-lease-claim-tokens.md`: step 2, which adds explicit claim tokens for the consumers this PR does not yet move. Consumer inventory (readers of `native_run_finalizations.lease_expires_at` that can act on a lapsed lease): - Restart recovery and takeover (`claimNativeRestartRecoveries`): claims first. It writes `recovery_state`, `lease_owner` or `controller_*` under the coordinator lock. - Finalizer and cleanup claims: claim first (`lease_owner`). - Failed-run owner release, environment settlement, observed-workspace preservation, retained-session cleanup: require a terminal run status, `committed`, or a recovery state. The rule refuses all of these. - Finalization reconciler, live recovery: require `retryable_failure`, a field the rule checks. - Board controller-stopped confirmation: writes `controller_host_hash`, which the rule checks. - Startup cleanup promotion: writes the run's `nativeStartupCleanup` marker, which the rule checks. It also applies only to runner-task leases. - Another server's orphan sweep: writes the run's ownership-unverified code. The rule checks it when the marker lands first. The reverse order is deferred (see Risks). - Workspace finalization copyback: acts on a lapsed post-result lease without a coordinator write. Post-result leases are therefore never renewed (deferred to step 2). - Accepted-result and token lease settlement, checkpoints, cancellation delivery, admission and the other readers: act on live leases only, or are read-only. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-execution-lease-reacquire.test.ts` passes (24/24) on this branch. With the source files from `origin/master`, 5 fail, including "lets the same owner take back a lease that lapsed untouched" and "renews from the shared timer after the sleep instead of interrupting the turn". - (a) The owner takes back its lapsed lease when no other actor claimed it. Also covered: the shared 5-minute timer, rows saved by the previous release, and the post-result relinquish. - (b) 14 cases where a consumer claimed first fence the owner out and leave the row unchanged. A release over a recorded recovery claim is refused. A different boot (a deploy mid-run), a different generation, a missing generation and a remote target are refused. - (c) Races have exactly one winner: a claimant and the owner lock the coordinator in each order, an in-flight run status write races the owner, and 8 simultaneous starts each have one winner. - Existing suites pass: `native-execution-lease.test.ts`, `native-session-executor.test.ts`, `native-runner-restart-recovery`, `native-runner-restart-lease-renewal`, `native-runner-restart-preparation` and `native-runner-restart-checkpoint-cost` integration tests. - Live proof: `B5 × TM2 --time-age-hours 1 --tasks 2 --seed 304` with `--server-from` this branch. Wave `w-20261110T083444-s304` passed 2/2, with no Retry and no `native_session_lease_lost`. - Both runs ran past the 300 s renewal after the one-hour aging, at the exact point where `origin/master` fails. t01 ran 310 s and t02 ran 319 s. Each was a single run that succeeded. - The harness used the aging exclusion from #20500 (merged): process start identity is not aged. Without that exclusion, TM2 also moves `controller_process_started_at`. The live owner then fails its identity check, which a real sleep never causes. An earlier wave on the unpatched harness, `w-20261110T072106-s304`, failed 0/2 for that reason only. ## Risks - Previous release: no persisted shape changes and no migration. Rows saved by the previous release have the same columns, and a null `controller_host_hash` is accepted. A deploy that lands mid-run cannot take back the old server's lease, because the boot and pid differ. - On the normal path, a live lease takes the same single coordinator lock as before. The run share lock is taken only after a lapse, and nothing is added before `turn.start`. - Deferred to step 2 (see the plan): - The orphan sweep of another server process can read the row before the owner takes the lease back and write after. That run would carry an ownership-unverified error while it is live. - Shared-workspace holder detection reads a silent run with lapsed leases as gone. - Post-result leases are not renewed. - Remote leases are unchanged. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code with tool use (shell, file editing, sub-agent code audit). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
|
0934b834c4 |
perf(workspaces): release retained cohort parent scratch early (#20447)
## Thinking Path > - Paperclip manages agent work and preserves its workspace files. > - Online migration captures local files and publishes remote workspace authority. > - Each capture verifies encrypted readback, restores the archive and compares the current source. > - A cohort kept every completed parent payload until the whole cohort published. > - A larger cohort could retain many complete copies while later members still worked. > - This pull request releases only finished parent payloads before a capture worker starts another member. > - Nested-child proof, final fresh checks and atomic publication keep their existing owners. ## Linked Issues or Issue Description Refs: #20418, #20437, #20441. **What happened?** A cohort retained each member's capture archive, verified archive, comparison archive and restored tree until atomic publication. The capture worker limit did not limit that retained parent data. **Expected behavior** Release completed parent payloads once their consumers finish. Keep files that a later publication check still needs. **Steps to reproduce** Run the added held-peer regression against the previous owner. It completes one parent, holds a later member and finds all four completed parent paths still present. ## What Changed - Release three named parent archives and the restored tree after a successful online cohort capture, before that worker takes another member. - Keep parent scratch roots and all nested-child verification areas. Keep every final fresh child and publication check. - Retain payloads for archived repository intents. Use the existing writable-tree cleanup for read-only restored directories. - Add real filesystem regressions for early release, retained child proof, late child mutation and the final archived-intent path check. Update the source-fence guide. This branch incorporates the final #20441 owner and both distinct saved-relative-path regressions. Its scratch-specific production change is nine added lines and one removed line. Single imports and saved journal/archive formats stay unchanged. ## Verification - Canonical Node24 preflight passed. The Runner build came from the content cache. After the current-master merge, the preflight checked 1,674 inputs and 1,817 outputs and found unchanged build inputs. Plugin SDK prerequisites passed. - Genuine RED on the unchanged #20441 owner: one held-peer case failed at four retained payload paths; the archived-intent control passed. The run took21.14s, with221 unrelated cases intentionally unselected. The first sandbox-restricted setup attempt hit DNS errors and was stopped; the normal network-qualified preflight then passed. - Initial treatment: four selected PostgreSQL/filesystem cases passed in38.61s, with219 intentionally unselected. The held peer observes released parent payloads and retained child areas. A later child mutation still refuses the whole cohort and publishes no authority or activity. - The archived-intent control observes all parent payloads at the actual final saved-relative-path check. The changed path refuses publication and keeps task/repository state intact. This exercises its actual singleton owner; archived-only sources do not enter the online cohort API. - After a normal merge of current master and the final #20441 test, the five affected cases and server TypeScript passed. No full local matrix was repeated. Initial CI failed the inherited recovered-upload assertion because the new retention test leaked its archive and filesystem spies. That case passed alone but failed in the adjacent three-case sequence (two pass, one fail); scoped `finally` cleanup made the same sequence pass three of three in17.15s, with191 unrelated cases intentionally unselected. No upstream assertion or production code changed. Server TypeScript passed again. All required [CI](https://github.com/paperclipai/paperclip/actions/runs/42746586511) passed on final normal-merge head `604c0f67bf7282ecd3f6e5e5da1c9d8637ef70cc` (56 success, four skipped and one neutral; no pending or failing checks). The previously failing server4/12 passed in10m23s. Fresh Greptile is5/5 on that exact head with no open threads. The merge resolves the duplicate #20441 squash insertion; both tested owner and test files are byte-for-byte identical to repair head `b5c8eed82b6a676985e1a8af785e5df2ae3be2ce`. The prior head received [Greptile5/5](https://github.com/paperclipai/paperclip/pull/20447#issuecomment-6757435885); its archived-cohort coverage warning was [answered using the actual public admission boundary](https://github.com/paperclipai/paperclip/pull/20447#discussion_r4520641017). - `git diff --check` passed. No hosted cleanup, migration, provider action or deployment was performed. ```sh node scripts/agent-preflight.mjs --skip-rust-debug pnpm --filter @paperclipai/plugin-sdk ensure-build-deps node --import ./cli/node_modules/tsx/dist/loader.mjs packages/db/src/test-postgres-command.ts pnpm exec vitest run server/src/services/remote-workspace-cohort-migration.test.ts server/src/services/remote-workspace-migration.test.ts -t 'releases completed parent payload|retains archived intent payload|shares the merged child verifier|rechecks the current archived repository intent path at publication' pnpm --filter @paperclipai/server exec tsc --noEmit ``` ## Risks - This reduces retained parent data. It does not establish a total disk bound. Active capture/encryption/readback files and retained child areas still need measured capacity. - Cleanup failure uses the existing worker abort and peer-join path before publication. Original source files, uploaded objects and journals remain intact. - Archived repository intents still need their restored payload at publication. Those payloads stay retained. No capture limit, budget or integrity guard changes. - Saved operations and archive identities carry forward unchanged. Resume still uses the existing full source and authenticated object checks. ## Model Used OpenAI GPT-6 (Codex), with tool use and code execution. The runtime does not expose a more specific model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
1b52f359dd |
test(restart-chaos): add config fault waves (CF1-CF3) and the CF4 design (#20446)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness (`tests/restart-chaos/`) starts an isolated control plane, runs real agent tasks and restarts the server during them > - Every wave starts from a new instance with valid default settings and one valid provider > - On 2026-11-09 production failed on state that a new instance never has: a setup command that named a renamed script (#20269), and an account default that did not exist (#20361) > - This pull request adds a "config fault" family (CF1–CF3): the harness breaks one saved reference on a task's own agent or project, restarts the server while the task is parked, repairs the setting and grades the result > - The benefit is that the harness now catches "stale saved setting" failures, including a run that retries forever or a task that stays Queued with no explanation ## Linked Issues or Issue Description Refs #20269 Refs #20361 **What existing behavior does this improve?** The restart chaos harness only tests new instances with valid settings. It cannot see failures that come from saved settings that point at a missing object. **Subsystem affected** `tests/restart-chaos/` (test harness only). No product code changes. **Current behavior** No wave breaks a saved setting. The CF2 failure (#20361) and the CF1 failure (#20269) were found in production, not by the harness. **Proposed behavior** `config-wave.ts` runs CF1–CF3. The oracle is not "the task completes". A task passes when all of these are true: - it completes, or it rests on a message that names the broken setting on the task page within a bound; - no run retries unchanged configuration forever; - no task stays Queued with no owner; - the rest survives a K1 or K2 restart; - after repair and the offered action (connection card, Retry, or agent setup Retry), the task completes. ## What Changed - `tests/restart-chaos/config-faults.ts` (new, pure): the catalog of broken references, the seeded plan, `parkState`, the task-level message check and `gradeConfigTask`. CF1 is a missing setup script. CF2 is a missing personal OpenRouter default. CF3 is a seeded walk over a deleted secret, a missing workspace path, a missing base ref, an unknown model, plus CF1 and CF2. `CONFIG_REFERENCES_NOT_BROKEN` lists the settings CF3 cannot break, and why. - `tests/restart-chaos/config-wave.ts` (new): the wave runner. Each task gets its own agent, credential secret and project. The runner breaks the setting through board routes, launches an A1 brief task, polls, fires K1 or K2 while the task is parked, repairs the setting, uses the offered action and grades the task. It reuses the existing supervisor, seed and A1 oracle. Waves are limited to 4 tasks and a concurrency of 2. The instance is deleted at the end. - `tests/restart-chaos/config-faults.test.ts` (new): unit tests for the plan, the arguments, `parkState` and the grader. These include the CF2 bug signature and the CF1 "message only on the run page" signature. - `tests/restart-chaos/README.md`: a new "Config faults" section and a row in the pieces table. - `doc/plans/2026-11-09-chaos-config-and-aged-state-faults.md`: the CF4 design (an instance aged by an older release, built on the existing K14 `--upgrade-from` machinery). CF4 is not built in this pull request. - There are no edits to `main.ts`, `types.ts`, `workloads.ts`, `invariants.ts` or `grader.ts`. Other agents work on those files in parallel. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 24 files, 291 tests pass (before the last unit test was added). The config-fault suite has 13 tests, and all pass. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: passes. - Live waves (local, isolated instance, one wave at a time): - **CF2 red before #20361** (base `8004ad069e`, before #20361 merged): - Waves `w-20261110T000602-s201-cf` and `w-20261110T004818-s202-cf` fail with `CF2/ai_default:bounded_rest+actionable_message+lifecycle_retrying_unchanged_config+queued_without_live_owner`. - The user sees `Queued`, a `resume` user action, and "Execution configuration check failed. Paperclip could not complete this step. It will retry automatically." There is no card. - A debug run with logging added to the lifecycle failure log (not committed) showed that the underlying error is `ai_connection_default_missing`. Verification retried it every 60 s, more than 20 times. - After the harness connects an account that is installed for the agent, the task completes 35 s later on its own. - The CF2 run after #20361, and the CF1 and CF3 waves on current master, are added below as they finish. ## Risks - Low risk. The change is test harness code and docs only. - A wave costs provider tokens only when the repaired task runs (one A1 brief). - The CF2 repair creates a real OpenRouter account in the isolated instance. The server validates the key live. - The oracle reads the task page through the API (interruption, execution status, cards, agent setup status and notices). It does not render the board UI. A message that appears only in the run's own error text is graded `message_only_in_run_detail`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code with tool use (shell, file edits, subagents). Standard context window. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
|
07f6e8ff75 |
perf(workspaces): group same-root cohorts with bounded workers (#20437)
## Thinking Path > - Paperclip preserves agent work when its control-plane server is replaced. > - Online migration imports local workspaces into authenticated snapshots. > - Several workspace records can point to one physical directory. > - Shared child verification avoids repeated work inside one held cohort fence. > - The eight-member limit splits 30 identical-root aliases into four cohorts and repeats the child passes. > - This pull request admits up to 32 members only for one canonical physical root, while keeping each work stage bounded to eight peers. > - The benefit is fewer repeated child checks while capture and publication guards stay in place. ## Linked Issues or Issue Description Refs: #19873, #19882, #20418, #20424, #20429. Related PR search found shared verification and restart fixes, but no larger same-root cohort owner. The shared verifier and archived-source changes are merged into this branch from current master. **Current behavior** Thirty fresh aliases of one physical root require cohorts of 8, 8, 8 and 6. Each cohort performs an initial and final local child-validation pass. The same children are checked eight times in total. **Proposed behavior** Admit one cohort with nine to 32 members only when every canonical resolved physical root is exactly equal. Keep the existing complete source-scope equality and admission guards. Keep mixed-leaf cohorts limited to eight members. Bound source reads, reservations, recovery checks, final readers, replay and cleanup to eight joined peers. Capture concurrency stays at most eight. **Reason and benefit** For N distinct canonical covered children, the 30-alias case changes 8N local child checks to 2N, a 75% reduction in repeated child validation. If N is 68, that is 544 checks versus 136. Thirty parent captures, uploads and restores remain required. No total wall-clock improvement is claimed. **Breaking changes** Saved cohort fields and exact membership remain unchanged. Older tools that enforce the eight-member bound refuse larger cohorts. Use the newer tool to resume them; never split or regroup their saved membership. ## What Changed - Raise the canonical service, saved-cohort and maintenance-request member bounds to 32. Keep the capture-worker bound at eight and single-workspace modes at one. - Require one exact canonical physical root for more than eight members. Keep the existing full source-scope comparison, leaf rule and P/I/L/F guards. - Use one private stage mapper with batches of at most eight. Join every started peer before an error can leave the stage. Stop future batches after a failure or cancellation. - Preserve the existing cancellation and pending-snapshot ordering for stages of at most eight members. Check cancellation around extended batches. - Keep all saved cohort, member, operation and snapshot identities, proof refresh and atomic publication checks. - Bound cleanup without skipping later scratch cleanup after an earlier cleanup error. - Extend the existing PostgreSQL fixture with ten real aliases, partial-upload resume, cold restoration, replay and mixed-root refusal. Extend the request boundary test. - Document old-tool refusal, work counts and the retained-scratch capacity bound. ## Verification - Canonical Node24 worktree preflight and Plugin SDK prerequisites passed. After merging current master, preflight passed again with unchanged cached build inputs. - Genuine RED on the old bound: both real ten-alias import/resume cases and the 32-member request reject the eight-member limit (3 failed; 47 not selected; 18.23 seconds). - First focused GREEN: 4 passed in 24.20 seconds. The fresh ten-alias case performs exactly four local child captures for two children: one initial and one final pass. Actual reader, transaction and upload maxima are eight. All ten archives cold-restore after the original source is parked; replay retains the same cohort. - The partial-upload case keeps all ten saved operation IDs and the existing snapshot IDs. More than eight members across different canonical leaf roots refuse before any journal or storage work. The request parser accepts 32 members and eight workers, and refuses 33 members or nine workers. - The first expanded corpus had 55 passes and one real compatibility failure: a new eager cancellation check skipped the old pending snapshot boundary. The mapper now retains existing cancellation ordering for at most eight peers. That failure remains recorded as a nonpass. - Final sources include current master at `18697b8d21`, with the verifier, archived-source and UI repairs. The selected four-file corpus passes 47 cases in 78.88 seconds, with 191 intentionally not selected. The full maintenance suite separately passes 23 cases in 10.21 seconds. Nine cases overlap; the final union is 61 distinct passing checks. - Final coverage includes all 27 cohort cases, all 23 maintenance cases, all six actual old-format compatibility cases and five archived prepared-source controls. Existing source mutation, finalizer contention, actual fence loss, cancellation peer joins, journal compare-and-swap and outer-commit rollback controls pass. - Final server TypeScript, module boundaries and whitespace checks pass. Only the resource-risk documentation changed after that test run; runtime and fixture bytes stayed unchanged. - Commands: `pnpm exec vitest run server/src/services/remote-workspace-cohort-migration.test.ts server/src/maintenance/online-workspace-migration.test.ts server/src/__tests__/online-workspace-migration-compatibility.test.ts`; archived controls use the existing `remote-workspace-migration.test.ts` cases named `unselected archived recipe` and `archived prepared source`; `pnpm --filter @paperclipai/server exec tsc --noEmit`; `node scripts/check-module-boundaries.mjs`. - Initial source review was 4/5 and requested a control for the extended reader-batch stop boundary. The test-only follow-up `ca80ae9cc8791a72dab632090488093c5d0916f3` adds actual ten-member failure and cancellation cases. They hold a first-batch peer, verify that the source fence stays held, and prove members nine and ten never start preflight. Both pass in 10.87 seconds with no new journal, snapshot or upload effects. Runtime source hashes are unchanged; the local union is now 63 distinct checks. - The first stop-test attempt observed the two controlled readers before all SQL-backed peers had entered the inspection callback. Its assertion was too early. The corrected test checks the held first-batch readers at that point, then requires all eight to join before completion. Both cases pass; the earlier attempt remains a nonpass. - Final head `ca80ae9cc8791a72dab632090488093c5d0916f3`: [required CI](https://github.com/paperclipai/paperclip/actions/runs/42737676664) and [aggregate verify](https://github.com/paperclipai/paperclip/actions/runs/42737676664/job/129575813765) passed. Full rollup: 56 successes, four intentional skips and one advisory neutral result. Classic Snyk passed. [Greptile](https://github.com/paperclipai/paperclip/pull/20437#issuecomment-6756419093) and [Ponytail](https://github.com/paperclipai/paperclip/pull/20437#issuecomment-6756388693) both reviewed that exact head at 5/5. The one review finding is addressed and resolved; no open findings remain. Older partial CI and the initial 4/5 review remain nonfinal evidence. Hosted acceptance and measured wall-clock savings remain pending. ## Risks - Completed parent captures retain their scratch until the atomic publication completes. A 32-member cohort can retain up to four times the parent scratch of an eight-member cohort. Check actual parent capture, verification and restore sizes against current target filesystem capacity before selecting a larger cohort. Keep the existing resource guards. - Larger member arrays must not raise execution fanout. Every reader/reservation/final stage is bounded to eight and joins current peers before failure. Capture workers remain at most eight; child-verification workers remain two. - More than eight members require the exact same canonical physical root and complete source scope. A different root, new writer, changed source, recovery owner or journal still refuses through the existing owners. - Shared verification is limited to the held invocation fence. The separate fresh final pass remains required after all captures join. Different child sets and restart attempts do not share a local-byte proof. - The old saved membership and archive identities remain authoritative. Older tools fail closed on the larger bound. No schema, archive or provider format changes are included. ## Model Used OpenAI Codex, GPT-6 family, with repository tools and code execution. The runtime does not expose a more specific deployment identifier or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
18697b8d21 |
perf(workspaces): share cohort child-source verification (#20418)
## Thinking Path > - Paperclip must preserve agent workspace bytes during online migration. > - A cohort can contain several aliases of the same physical source. > - The importer verifies imported children before capture and again before publication. > - The immutable-object cache avoids repeated downloads but still recaptures local child bytes for every alias. > - This pull request shares the initial proof within the held cohort fence and checks fresh bytes once more after every capture joins. > - The benefit is less repeated local archive work while child mutations still block the whole cohort. > - Required browser CI twice reproduced an independent stale composer-mode bug. A focused regression proves that refreshed persisted task mode was ignored after mount. The state repair follows task data when there is no local override. Explicit local mode choices retain their own generation so a completed send cannot clear the next draft’s choice. ## Linked Issues or Issue Description Refs: #20311, #20387, #20401, #20404. **Current behavior** Two aliases recapture the same imported child's local bytes four times, even with one immutable object read. A larger alias cohort repeats this multi-GB work for each member. **Proposed behavior** Share the successful initial verification for the same physical root, Git roots and exact recorded child set or discovery mode. After all captures join, run a separate fresh final verification for each root and recorded child set. Keep every member's database and publication checks. **Reason and benefit** The local PostgreSQL fixture proves that two aliases now perform two child recaptures instead of four. Hosted timing improvement has not been measured. ## What Changed - Add invocation-local initial and final proof maps to the existing cohort child verifier. - Retain the existing canonical source-owner check before and after shared results. - Defer cohort final nested checks until all captures finish, and join every final reader before publication or fence release. - Keep singleton verification, saved journal schemas, snapshot IDs, archive bytes and per-member publication checks unchanged. - Extend the existing real PostgreSQL alias fixture with a child mutation during the last parent upload. - Repair the required CI planning-mode race with a nullable local override: absent override follows task data; a user selection belongs to the draft. A successful send or confirmed resend/receipt clears it only if no newer choice exists. Failed saves preserve the mode and restored message. Queued-edit backup retains whether the mode was explicit. Leave the browser assertion and timeouts unchanged. ## Verification - Canonical Node24 worktree preflight and plugin SDK prerequisites passed. Server TypeScript passed. - Genuine unchanged-source PostgreSQL RED: two aliases performed four local child recaptures instead of two (one failed, 22 intentionally not selected). GREEN: five source-sharing, last-upload child mutation, capture/cancellation join and actual outer-commit rollback controls passed (18 intentionally not selected; 66.53s). The actual fence-loss control also passed (one passed, 22 intentionally not selected; 38.13s). Both wrappers completed cleanup. - Migration source and tree bytes remain identical to locally qualified `466837cfe997924c3794fd5d0e96134cab1cd04c`; later commits only repair the composer and its tests. - Required planning browser CI originally failed, then failed its one unchanged-head retry, because the planning chip remained absent after opening-question dismissal. Those failures remain recorded ([original](https://github.com/paperclipai/paperclip/actions/runs/42723622165/job/129523785985), [retry](https://github.com/paperclipai/paperclip/actions/runs/42723622165/job/129534415590)). The final-head identical case passes in 30.5s, with all 34 shard cases passing ([final CI](https://github.com/paperclipai/paperclip/actions/runs/42731309796/job/129550549775)). Assertions and timeouts were not relaxed. - Composer regressions reproduced stale loaded mode, optimistic-save rollback, overwritten next-draft choices, and stale mode after confirmed resend/receipt. All 145 composer tests now pass (2.88s). Canonical UI TypeScript and token gates pass. The original planning browser case also passed locally (15.5s; 26.9s including setup). - The first final-head serialized-suite attempt was automatically cancelled at its unchanged 20-minute job limit. Grouped Turbo output did not expose an individual failing assertion; cause remains unknown ([retained attempt](https://github.com/paperclipai/paperclip/actions/runs/42731309796/job/129550550373)). One same-head diagnostic retry was a cache miss and executed all 419 heartbeat recovery tests successfully in 376.94s on the identical CI checkout ([retry](https://github.com/paperclipai/paperclip/actions/runs/42731309796/job/129561196792)). It is not classified as infrastructure or a known flake. - Final exact head `ba1f85f18085dd0e84c594ed0cb68d5bf817c93c`: all 61 checks terminal (56 success, four skipped, one neutral); required CI is green ([workflow](https://github.com/paperclipai/paperclip/actions/runs/42731309796)). Fresh Greptile is 5/5 with every review thread resolved ([review](https://github.com/paperclipai/paperclip/pull/20418#issuecomment-6754705323)). Focused commands: ```sh node --import ./cli/node_modules/tsx/dist/loader.mjs packages/db/src/test-postgres-command.ts pnpm exec vitest run server/src/services/remote-workspace-cohort-migration.test.ts -t "shares the merged child verifier|joins active capture peers|keeps every authority and activity unpublished" pnpm exec vitest run ui/src/components/task-chat/TaskChatComposer.test.tsx pnpm --filter @paperclipai/server exec tsc --noEmit pnpm --filter @paperclipai/ui typecheck pnpm check:token-gates ``` ## Risks - Reuse is limited to one invocation under the same complete source and finalizer fences. Different physical roots, Git roots, recorded child sets and discovery mode have different keys. No local-byte proof is stored durably or reused after restart. - Initial and final maps are separate. A child change during any parent upload must fail the fresh final verification before the atomic publication transaction starts. - Final readers all finish before an error can release the source fence. Per-member database child rows, source fingerprints, journals, keys and publication locks retain their existing checks. - This reduces repeated verification for aliases. It does not change capture concurrency, upload policy or immutable checkpoint validation. Hosted speedup remains unverified. - The CI repair must distinguish task data from explicit draft intent. Only a successful send with the same choice generation clears its override; a later draft selection and failed-save rollback retain it. Regression tests cover these transitions. It adds no styling, request, or persistence changes. ## Model Used OpenAI GPT-6 (Codex), with tool use and code execution. The runtime does not expose a more specific model identifier or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
bf818280f2 |
perf(workspaces): reserve cohort members concurrently (#20401)
## Thinking Path > - Paperclip controls agent work and preserves its workspaces. > - The online importer moves up to eight shared-source leaves under one source fence. > - Each member must have a durable reservation before capture starts. > - Serial reservations repeat the same validation cost before any archive work. > - This pull request reserves members in parallel through their existing independent transactions. > - Every reservation finishes before an error can release the fence or start capture. ## Linked Issues or Issue Description Refs: #19873, #20387, #20394. **Current behavior** A measured six-member cohort spent about 110 seconds in serial reservations. Each reservation keeps its existing source, task, recovery and journal checks. **Proposed behavior** Start the bounded member reservations together. Join all outcomes, retain member order, and refuse capture if any member fails. Keep the exact partial-journal resume contract. **Reason and benefit** This removes serial metadata work before capture. Hosted speedup has not been measured for this change. ## What Changed - Replace only the cohort reservation caller loop with `Promise.allSettled`. - Preserve the existing per-member transaction, preassigned operation ID, source fingerprint and all source fences. - Extend the real PostgreSQL failure fixture to require overlapping reservations and complete peer closure before fence release. - Document the unchanged partial-journal and atomic publication contract. ## Verification - Canonical cached worktree preflight and plugin SDK prerequisites passed on Node 24.22.2. - The unchanged production baseline failed the new reservation overlap assertion: one reader entered instead of two. The canonical PostgreSQL wrapper cleaned up. The other 21 tests were intentionally not selected. - The first baseline command failed before PostgreSQL because root-level `tsx` resolution was absent. The corrected command uses the checkout's explicit `cli/node_modules/tsx/dist/loader.mjs`. - The four focused PostgreSQL controls passed before the final rebase: overlap/join failure, exact partial-journal resume, parallel capture with both cold restores, and atomic outer-commit rollback. Eighteen tests were intentionally not selected. Server TypeScript passed. - The final branch is based on merged #20394 at `302ada90dfda0e8fbcd8a514d506d310c9843bcb`. The two directly affected PostgreSQL reservation/resume controls passed again on final head `06d436e45705da3b616a2b91725960f6e0ea52c0` (2 passed, 20 intentionally not selected; 100.56 seconds). Server TypeScript passed again. The canonical wrapper completed cleanup. - The separate Ponytail review remains 4/5. Its nonblocking fan-out warning was resolved as the documented phase contract; no source change was required. Optional multi-error aggregation is outside this change. - Fresh Greptile review is 5/5 on `06d436e45705da3b616a2b91725960f6e0ea52c0`, with no actionable findings or unresolved threads: https://github.com/paperclipai/paperclip/pull/20401#issuecomment-6754025480. - Exact-head CI is complete: 56 success, 4 skipped and 1 neutral, with no failed or pending checks. Aggregate `ci / verify` passed: https://github.com/paperclipai/paperclip/actions/runs/42718439355. - Reproduce the final focused cases on Node 24: `node --import ./cli/node_modules/tsx/dist/loader.mjs packages/db/src/test-postgres-command.ts pnpm exec vitest run server/src/services/remote-workspace-cohort-migration.test.ts -t "joins concurrent reservation validation readers|resumes the exact membership after reservation stops halfway"`. Run `pnpm --filter @paperclipai/server exec tsc --noEmit` for TypeScript. ## Risks - API phase contract: `concurrency` limits capture workers. Preflight, reservation and post-finalizer validation fan out over the exact selected member set, with the canonical maximum of eight. The original DB pool bounds connections. `concurrency: 1` does not promise one metadata query or transaction at a time. - Up to eight reservations can use independent pool connections at once. Members that pass can leave durable journals when a peer fails. The existing exact-member resume path retains those journals; this change does not retry automatically. - Capture and publication remain blocked until every reservation settles. Publication stays serial in one transaction. Saved journal schemas, singleton behavior and recovery proof checks do not change. - This does not change concurrency inside the publication transaction. One PostgreSQL connection serializes that SQL, and changing its row-lock order has no measured benefit here. ## Model Used OpenAI GPT-6 (Codex), with tool use and code execution. The runtime does not expose a more specific model identifier or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
9c634e42e4 |
fix(workspace): bound queued source-lock admission for imports (#20370)
## Thinking Path > - Paperclip manages AI agents and their work. > - Existing local workspaces need durable imports before a volume-free deployment. > - Imports must acquire the same source locks as workspace writers. > - Try-only admission can repeatedly lose to background maintenance, even when each writer finishes. > - This pull request adds an optional bounded wait to the canonical migration APIs. > - Operators can join the source-lock queue while all existing preservation checks remain active. ## Linked Issues or Issue Description **What happened?** An online import can repeatedly return `workspace_source_busy` while background workspace maintenance reacquires its project source lock. The import does not join the PostgreSQL lock queue. **Expected behavior** An administrative import can wait for a finite writer to finish. One wait budget covers the complete source scope. Timeout or cancellation must release acquired locks before the import reserves a journal. **Steps to reproduce** 1. Hold the source lock for an eligible local workspace. 2. Start an online import while the writer holds that lock. 3. Release the writer. Before this change, the import has already refused admission. With `sourceLockWaitMs`, the import acquires the source scope and commits normally. **Paperclip version or commit** Reproduced on master `0e3fdebea8257bdff11573081c76b306f12db230`. **Deployment mode** External PostgreSQL with an online workspace import. Related preservation work: https://github.com/paperclipai/paperclip/pull/20311. ## What Changed - Add optional `sourceLockWaitMs` to singleton and cohort migration APIs. Omitted options keep current admission behavior. - Queue the sorted source keys with one decreasing wait budget. Abort cancels only the dedicated backend identified by PID and backend start time. - Use an independent source transaction for queued singleton imports. Keep lifecycle checks, fresh source checks, journal ownership, and publication after commit. - Document the canonical API option. Add real PostgreSQL importer regressions for release, timeout, abort, and partial-key cleanup. - Correct the inherited task-case lookup summary so the existing search-discovery regression passes without changing its assertion or search algorithm. ## Verification - Baseline RED: the new singleton regression fails on the unchanged product because the import returns busy without queueing. - `pnpm exec vitest run server/src/services/remote-workspace-migration.test.ts -t 'queued|bounds the whole source queue'`: 20 passed, including five new cases. - `pnpm exec vitest run server/src/services/isolated-remote-workspace-source-lock.test.ts`: 4 passed. - Server layer typecheck, `node scripts/check-module-boundaries.mjs`, and `git diff --check` passed. - Tests used Node 24.22.2 and the existing embedded PostgreSQL fixture. Hosted rollout verification remains pending. - CI build, typecheck, browser suites, all serialized suites, and the other general shards passed. The final server shard failed only the inherited `case-history-api-discovery.test.ts:14` search-ranking assertion. Its unchanged local regression reproduces on the base and passes 3/3 with a separate one-line route-summary correction. The shared one-line summary correction is included in follow-up commit `8163ffa9d4728134676b84c8c242a5bf62c802b5`; the unchanged case-discovery suite passes 3/3 locally. The original CI attempt remains failed: https://github.com/paperclipai/paperclip/actions/runs/42704666536/job/129458605783. ## Risks Queued admission can hold earlier source keys while it waits for a later key. The caller controls the finite budget. The abort path opens a short-lived dedicated cancellation connection. If cancellation cannot connect, PostgreSQL's remaining lock timeout still bounds the wait. This option must be called at the root of administrative work; an external scope that already holds the same source locks can cause self-contention. The existing pre-held journal-resume path must keep its omitted-option behavior. ## Model Used OpenAI Codex, based on GPT-6, with tool use and code execution. The exact deployment identifier and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes: #` / `Refs: #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
7f55323abb |
fix(recovery): end a remote workspace epoch whose writer died during preparation (#20346)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - A task with a remote workspace prepares that workspace before the
run selects its runtime, and preparation marks the workspace epoch dirty
with the run as its writer
> - If the server is killed during preparation, the run never selected a
runtime and never started provider work, but the dirty epoch remains
> - Failed-workspace preservation, the path that resolves a dead
writer's dirty epoch, applies to native runs only, so nothing owned this
epoch
> - The dirty epoch kept the dead run's sandbox lease as a pending
export and held the workspace against the task's retry, so the task
never recovered
> - This pull request lets the ownership reconciler end such an epoch at
the last committed save
> - The benefit is that a restart during workspace preparation no longer
wedges the task
## Linked Issues or Issue Description
**What happened?**
In a restart-chaos wave on Daytona (a remote workspace that clones a
real repository), the controller was killed while each run's first
workspace save was in flight. That save is now settled by #20326. Each
run ended `process_lost` with a bounded retry, and both runs were still
in `execution_stage: preparing` with `runtime_mode_resolved_at: null`.
The workspace authority stayed `dirty: true`, `snapshotId: null`, with
the dead run as writer. The dead run's environment lease was proven
owner-gone but was refused by its settle as `workspace_export_pending`,
and the reconciler reported `I2_settled` violations. The retries stayed
`queued` until the test harness deleted the environment 21 minutes
later. Then they failed `remote_workspace_execution_required`, because
the environment was gone.
**Expected behavior**
When a writer dies before runtime selection, its unsaved files are only
the restored source and preparation output, because no agent turn ran.
Recovery must end that epoch at the last committed save, so the lease
settles and the retry restores the workspace into a new directory. A
writer that reached runtime selection must keep its existing recovery.
**Steps to reproduce**
1. Run a task whose agent uses Paperclip Runner with a remote sandbox
workspace.
2. Kill the server while the run is still preparing the workspace (for
example during its first save).
3. After the restart, the run ends `process_lost` and its retry is
queued, but the workspace stays dirty and owned by the dead run, the
lease stays active, and the retry never starts.
**Paperclip version or commit**
`master` at
|
||
|
|
c67909cb0f |
fix: keep tool approvals valid through OAuth refresh and delayed review (#20340)
## Thinking Path > - Paperclip lets people manage agent work and approve app actions. > - An approval must bind the action that the person reviewed. > - OAuth access tokens can expire while that person is away. > - The old target snapshot treated each token refresh as a new authorization. It also imposed a one-hour review deadline. > - This pull request separates token renewal from authorization identity and keeps the human decision durable. > - Current access, the reviewed arguments, meaningful target changes, task closure, and execution receipts still control dispatch. > - People can approve later without repeating a decision after routine connection maintenance. ## Linked Issues or Issue Description **What happened?** A pending app action returned `approved_tool_target_changed` after its OAuth tokens refreshed. The tool and connection settings were unchanged. The same card also expired after one hour. A sign-in problem could discard an accepted decision before provider dispatch. **Expected behavior** The original card remains reviewable after routine OAuth refresh and a server restart. A later approval runs the exact reviewed action once. A connection repair preserves the decision while Paperclip verifies the same authorization. Meaningful changes require a new review. **Steps to reproduce** 1. Connect an OAuth MCP app and configure an action to require approval. 2. Request the action, then refresh its access and refresh tokens before approving. 3. Approve the original card. Before this change, it returns HTTP 409 even when the authorization is unchanged. 4. Wait more than one hour. Before this change, the card expires without a task-specific deadline. **Paperclip version or commit** Reproduced from master at `76254ead2229`. The final branch is rebased onto current master. ## What Changed - Add versioned approval target comparison. New snapshots bind credential identity epochs, consent, permissions, ownership, connection configuration, tool contracts, and credential references. Authenticated OAuth refresh preserves identity. Manual credential replacement does not. - Remove the implicit one-hour deadline from ordinary app action reviews. Keep explicit deadlines and terminal-task expiry. Synchronize the nullable interaction deadline in shared contracts and UI. - Keep accepted, undispatched actions when sign-in is required. Show a saved-approval repair state. Resume on committed connection changes after target and access revalidation, without an idle retry loop. - Preserve the existing execution claim, result receipt, and uncertain-outcome no-replay rules. - Explain meaningful target changes without inviting the person to retry the stale card. - Add a production task-thread Storybook example and document compatibility and recovery behavior. ## Verification - Red: production-refresher invalid-grant handling and formal-approval withdrawal during a sign-in wait also reproduced as failing regressions before their fixes. - Red: before implementation, the local OAuth HTTP fixture failed for both public and confidential clients with `approved_tool_target_changed` after token refresh. - Green: the extended fixture passes for public and confidential clients with personal and shared grants. It covers actual local HTTP token exchanges, multiple refreshes, a rebuilt gateway, an ended originating run, approval after 25 hours, duplicate approval, preserved sharing arguments, sign-in waiting, and event-driven repair. The gateways use the production OAuth refresher; an invalid refresh preserves the accepted decision before provider dispatch. Omitted-scope responses cover equivalent scope-reporting changes. The provider is an isolated local fixture, not a deployed Notebook service. - Negative controls refuse changed consent, scopes, tool definition, destination, and manual credential replacement before provider execution. - Existing gateway service, connected-tool acceptance, secret owner, secret service, unsigned-review queue, and connection-health suites passed. Explicit-deadline fixtures still prove expiry behavior, including deadlines on accepted sign-in waits. A withdrawn formal approval cancels a waiting action before connection recovery. - Approval card, task-chat, error-copy and target-comparison tests passed. Server and UI typechecks passed. Token gates and Storybook catalog checks passed. - Inspected the new saved-approval repair card in local Storybook using the production task thread. It shows a reconnect link and no second approval button or false running state. - `pnpm verify --explain` returned exit 2 and requested broad CI coverage. [Full CI](https://github.com/paperclipai/paperclip/actions/runs/42677855507) passed on `eb4f1c7caa9ee6e802c314ffc5eadec475a5de0c`, including build, typecheck/release registry, general and serialized tests, and all eight browser shards. - Greptile reviewed the same final head at 5/5. Both findings from the first review are fixed and their threads resolved; no open follow-ups remain. ## Risks - No database migration is needed. The new comparison reuses the vault identity epoch. Only authenticated OAuth refresh code can preserve that epoch; HTTP credential-write schemas cannot select that behavior. - Earlier signed requests keep their original exact-version checks and saved deadlines. Already-expired cards are not revived. A historical card whose credentials already changed needs a fresh review. - A new consent, account, permission set, destination, or manually replaced credential must not inherit the old authorization. Reconnection continues automatically only when revalidation proves the same saved authorization. - The interaction enrichment now permits a null deadline. Existing string deadlines remain valid. The web client and shared validator change together. - This PR does not deploy to staging or perform a live Notebook publication. ## Model Used OpenAI Codex, GPT-6 family, with repository tools, code execution, and browser inspection. The runtime does not expose the exact model ID, context-window limit, or reasoning-mode label; those values are not asserted. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> |
||
|
|
251c6d5c69 |
fix(recovery): settle a remote workspace save whose process died mid-save (#20326)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs in remote sandboxes save the workspace as snapshot
operations (`workspace_operations`, phase `remote_workspace_snapshot`).
Each save invocation is listed in `saveWork.activeIds` while it runs
> - When a controller process is killed during a save, its invocation
stays listed and the operation stays `running`
> - Nothing recorded which process ran the invocation, so the shared
ownership reconciler could not prove the owner gone and never settled
the save
> - The task's bounded retry then waited behind that save forever
> - This pull request records the owning process for each save
invocation, and lets the reconciler settle a save whose owner is proven
gone
> - The benefit is that a restart during a save no longer wedges the
task
## Linked Issues or Issue Description
**What happened?**
In a restart-chaos wave on Daytona with a real cloned repository, two
tasks each started their first remote snapshot save (`boundary:
initial`, epoch 1). A SIGKILL hit the controller during the save, before
any upload. Each run ended `process_lost` and got a bounded retry. The
retries stayed `queued` forever, the snapshot operations stayed
`running` with `saveWork.activeIds: [<dead invocation>]` and
`quiescedAt: null`, and the sandbox leases stayed active. The ownership
reconciler reported the saves as owner-gone but had no way to settle
them.
**Expected behavior**
After a restart, recovery must settle a remote snapshot save whose
owning process is gone. It is marked failed with provenance, so the
task's retry can continue. A save that is still running must stay
untouched.
**Steps to reproduce**
1. Run a native task with a remote workspace (for example Daytona with a
`git_repo` project source).
2. Kill the controller (SIGKILL) while the first workspace save is in
flight, before its upload.
3. After the restart, the run ends `process_lost` and its retry is
promoted to `queued`, but the save operation stays `running` and the
retry never starts.
**Paperclip version or commit**
`master` at
|
||
|
|
1b7fe33432 |
fix(workspaces): preserve import verification across process restarts (#20311)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Online import preserves old local files before the web server can use a disposable filesystem. > - A parent import must authenticate saved child archives before it can omit their directories. > - The operation journal preserves import identity, but completed child checks existed only in process memory. > - A container restart therefore repeated downloads, decryption and restoration that had already succeeded. > - This pull request saves a versioned verification receipt on each immutable child snapshot. > - A replacement process can reuse those checks while it reacquires locks and verifies current local files. ## Linked Issues or Issue Description Refs #19854 for verification reuse within a live source fence. Refs #19873 for cohort import. Refs #20298 for the separate remote-hold admission fix. Searched related workspace verification and restart PRs; none provides durable child verification. **What happened?** Replacing a server container interrupts a long online import. The original operation and snapshot survive, but a later invocation downloads, decrypts and restores completed child archives again. The current cache ends with the source-lock owner. **Expected behavior** Completed immutable checks survive process and scratch loss. Resume keeps the same operation and snapshot, reuses only correctly bound verification, and publishes once. Current local files and Git state must still be checked under fresh source locks. **Steps to reproduce** 1. Import two nested child workspaces, then start the parent import. 2. Kill the importer after a child verification finishes. Remove its scratch directory. 3. Resume in a new process. Before this fix, the completed child is downloaded and restored again. **Paperclip version or commit** The regression fails on the previous migration verifier. The final branch includes master at `189fefb99183953be37f0d1084275fbb0f35d6e2`. **Deployment mode** Managed Linux server with PostgreSQL and explicit S3 storage configuration. ## What Changed - Add optional opaque storage namespace identity. S3 binds it to the explicit endpoint, region, bucket, normalized prefix and addressing style. Implicit endpoints retain full verification. - Save a versioned child-snapshot receipt only after canonical crypto, manifest and file restoration succeed and a second HEAD confirms the same strong ETag and size. - Bind reuse to the complete snapshot reference, key material, child lineage, source operation and storage namespace. Require a fresh strong HEAD and canonical validation that the current secret provider can unwrap the saved snapshot key. - Keep current directory/Git capture and source-owner checks on every invocation. Read-only verifier callers do not write checkpoints. - Merge checkpoint metadata under a short row lock without changing parent journal hashes, snapshot retention or update timestamps. Bound reads and writes. Lock timeouts, statement timeouts, deadlocks and serialization conflicts are safe cache misses. Other errors remain fatal. - Extend real PostgreSQL/Git/archive tests with process death, scratch removal, resume, changed bindings and single publication. ## Verification - Baseline RED: two import invocations read each child twice. With durable receipts, each child is read once across both invocations. - Real process regression passed on the final rebased head: SIGKILL the owned importer after one checkpoint, delete all scratch, and resume in a new Node process. It retains the same operation and snapshot, reads each child once across both processes, and records one publication. - 53 focused cases passed in 104.86 seconds on the final production bytes before rebase. They cover existing nested/cohort behavior, namespace identity, changed source/reference/key/lineage/ETag, unknown proof versions, weak validators and bounded row contention. - After clean rebase and stronger scratch/snapshot assertions, the real process case passed again in 18.54 seconds. Server TypeScript, module boundaries and diff checks passed on final head `9da68ee7ce4a778201ef85e21a11c49eeea7bc16`. Required CI supplies final-head broad coverage. - The initial restricted-network dependency setup failed; unchanged preflight passed with normal network access. Initial TypeScript failures in test pipe assertions were corrected. No failed attempt is counted as a pass. - Review found a missing current-master-key check. A real restart regression with all child checkpoints complete reproduced unsafe publication after switching to a different valid master key. Follow-up `7ea92c2abd9ffdb2e7dd81cecf62c8fc7a6e74a5` delegates to the canonical snapshot-key owner before both durable and in-memory cache hits. - The fixed process regressions passed 2/2. Ten key-authority and existing identity/source guard cases passed. The wrong-key process now refuses with zero additional child payload GETs, no publication, and unchanged original journal/snapshot rows. Server typecheck, module boundaries and diff checks passed on the follow-up. - Review also found that transient database failures could make the optional checkpoint cache stop a valid import. Read and save now fall back only for SQLSTATE `55P03`, `57014`, `40P01`, and `40001`. Connection loss, identity changes, crypto failures, programming errors and cancellation remain fatal. Reads use a read-only transaction with one-second lock and eight-second statement limits. - The checkpoint database-fault regression failed two cases before the fix. All four recoverable import cases then passed. The final selected run passed 21 cases, including fatal-error and cancellation controls, real row contention, both process-restart cases and key/identity guards. Server typecheck and boundaries passed. Fault injection is limited to the checkpoint database boundary; the importer, PostgreSQL, Git, archives, storage and publication remain real. - Latest master was merged without rewriting published history. Six real import/process-restart cases passed again on final head `8c8e4213a680ebb5c8d6864416bc3a4f271e1e76` in 38.03 seconds. [Required CI](https://github.com/paperclipai/paperclip/actions/runs/42658248222) finished with 54 successful checks and four intentional skips; Snyk passed. [Fresh review](https://github.com/paperclipai/paperclip/pull/20311#issuecomment-6745354381) is 5/5 with all four threads addressed and resolved. Hosted verification after normal release delivery remains pending. ## Risks - A stale receipt could skip authentication of a different object. Complete reference/lineage/namespace binding, fresh strong HEAD and row-locked save validation prevent that reuse. Missing or unsupported hints trigger full verification. - Mutable local files can change between attempts. Their source locks and fresh captures remain required; a checkpoint never certifies current local bytes. - Checkpoint persistence can contend with other snapshot owners. It has one-second lock and eight-second statement limits; the four classified transient SQL errors retain the normal fully verified path. Other database errors stop the import so lost source ownership cannot be hidden. - No schema, archive format, public API or saved-run identity change. Older snapshots without receipts remain readable and get full verification. Existing data and recovery obligations stay intact. ## Model Used OpenAI Codex, GPT-6.1 Sol (`gpt-6.1-sol`), with reasoning, repository tools and code execution. Context-window size is not exposed by the runtime. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge |
||
|
|
099adb2728 |
fix(recovery): settle a remote workspace save that stopped without finishing (#20291)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs in remote sandboxes save the workspace as snapshots.
Each save is a `workspace_operations` row with phase
`remote_workspace_snapshot`
> - The ownership registry lists that kind as in-progress work, but it
had no settle step
> - When a restart cut off a failing final save, the row stayed
`running` after its invocations had all stopped and its run had ended
> - The retained-source fallback refuses while any workspace operation
is running, so a task whose saved files could not be verified could
never continue from its last save
> - This pull request lets the ownership reconciler settle that row as
failed
> - The benefit is that the automatic fallback can continue such a task
without any user action
## Linked Issues or Issue Description
**What happened?**
A native run in a remote sandbox finished its turn. Its final workspace
save failed because the sandbox disk was full ("No space left on device"
in the snapshot copy). A graceful restart then interrupted the retry.
The run ended as `native_workspace_sync_out_retry_exhausted`. Its
`remote_workspace_snapshot` row kept `status: running` for six days,
with `saveWork: { activeIds: [], unknown: false, quiescedAt: <time> }`
and `exportCleanup: { state: "pending" }`. The retained-source fallback
checked the task and refused with `workspace_operation_running`, so the
task stayed blocked.
**Expected behavior**
A save with no active or unknown invocation, whose run is terminal, is
finished work. The reconciler must mark it failed, so the fallback can
continue from the last committed save. A save that might still be
running must stay untouched.
**Steps to reproduce**
1. Run a native task in a remote sandbox. Make its final save fail, then
restart the server during the retry, so the save's work is quiesced and
the operation stays `running`.
2. Let the task's recovery refuse with an unverifiable-source failure
(for example `allocation_ownership_ambiguous`).
3. The retained-source fallback refuses with
`workspace_operation_running` on every check, and the task never
continues.
**Paperclip version or commit**
`master` at
|
||
|
|
5c73633a8f |
fix(workspaces): preserve remote recovery holds during online import (#20298)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Online workspace import preserves old controller-local files while unrelated remote work continues. > - A task can have an old local checkout and a separate durable remote workspace. > - A terminal remote run can retain a file-recovery hold on that task. > - The legacy source checker rejects remote references, so that hold blocks independent local imports. > - This pull request proves that the hold belongs only to the separate remote source and fences its finalizer during import. > - Old files can migrate while the remote recovery obligation stays intact. ## Linked Issues or Issue Description Refs #19873 and #19854 for the existing cohort and source-verification paths. Searched related online migration and remote recovery PRs; no duplicate fix was found. **What happened?** Online import rejects a legacy Git checkout with `online_workspace_source_scope_unproven` when a same-task failed run has a prepared reference to a separate durable remote workspace. The legacy run reader rejects remote references before the hold can be classified. **Expected behavior** Independent local files can import under their source fences. The separate remote workspace, dirty writer, allocation, accepted result and recovery hold must stay intact. **Steps to reproduce** 1. Create related local Git workspaces. 2. Record a terminal failed native run and an exhausted file-recovery hold for a separate durable remote workspace on the same task. 3. Import the local leaf online. The old checker refuses before capture. **Paperclip version or commit** Reproduced on master at `052c638b519ebb15f81e4fcd1a96adf6cd3b2f10`. **Deployment mode** Managed server with PostgreSQL and remote Runner execution. ## What Changed - Add a distinct remote-only preservation proof. Keep remote runs excluded from legacy export sources. - Use the canonical source inspector and prepared-reference matcher. Bind the run, finalizer, action, task, accepted result, authority, allocation and prior committed snapshot. - Acquire the held run's finalizer fence through the existing sorted lock path. Existing source-proof reinspection rejects changes before publication. - Preserve previous-release prepared references with an older baseline snapshot. Do not require that snapshot's epoch or source run to equal the current run. - Add real PostgreSQL import, authenticated cold restore, preservation, refusal, lock-contention and publication-race regressions. Document the behavior. ## Verification - Genuine baseline failure on unchanged product: the real importer refuses with `online_workspace_source_scope_unproven`. - Final focused regression file: 23 tests passed in 13.19 seconds through the canonical isolated PostgreSQL wrapper on Node 24.22.2. - 24 existing affected migration/recovery tests passed against the same final production bytes. Another 83 tests were explicitly excluded by the focused selection; no full-suite pass is claimed. - The positive imports local bytes, confirms the remote finalizer fence is held during upload, leaves all remote records unchanged, parks the original repository and authenticates/restores the archive. - Refusal and race cases cover malformed, foreign, same-local and mixed references; active owners/writers; changed allocation, result, task, action, authority and baseline; and finalizer contention. - Server TypeScript check, module-boundary check and `git diff --check` passed. - An intermediate run had one incorrect assertion that expected an explicit empty legacy-source array. The canonical journal omits that empty field. The final assertion accepts absence and still forbids any exported remote source. Final relevant results are 23 new plus 24 existing passes, rather than a claim that the earlier invocation passed in full. - Required CI passed on head `6a278e717db2b5f614b0a595f99bfd0fb5430227`: 55 successful checks and 4 intentional skips. [CI run](https://github.com/paperclipai/paperclip/actions/runs/42650025033). Greptile reviewed that head at 5/5 with no actionable findings or open inline/review comments. Hosted continuation remains pending normal delivery. ## Risks - A false disjointness proof could admit an unsafe local import. Strict company/task/source binding, the canonical inspector, finalizer fencing and publication revalidation retain refusal on uncertainty. - This does not recover or settle remote files. Dirty authority and the original recovery obligation remain unchanged. - No schema, public API, archive format or saved-run identity changes. Existing persisted state carries forward. ## Model Used OpenAI Codex, GPT-6.1 Sol (`gpt-6.1-sol`), with reasoning, repository tools and code execution. The runtime does not expose the context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge |
||
|
|
ac0d0374f2 |
perf(issue-detail): measure task opens on real instances (#20276)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - People open a task from their issues list to see what an agent is doing and to reply > - On a busy staging instance, the composer and the latest messages take 1.1–2.0 s to appear after a click. For one task with a slow run log, they took 13.9 s > - Local fixtures do not reproduce this. They do not have real network round trips, real server contention, or real object-store latency > - This pull request adds the measurement we need: a browser harness that times a task open on a real instance, and a Server-Timing entry on every API response that separates server time from network time > - It also records the staging baseline and the improvement plan > - The benefit is that each later performance change can be measured on real data with the same tool, before and after deploy ## Linked Issues or Issue Description **Problem or motivation** Opening a task from the issues list is slow on instances with real data. The composer and the latest chat messages can take seconds to appear. We could not measure this on staging in a repeatable way. We also could not tell server time from network and queueing time for most API requests. **Proposed solution** Add a reusable browser harness for task opens, and add a `paperclip_api` Server-Timing entry to every API response. **Alternatives considered** Local Playwright fixtures only. They do not reproduce real network round trips, server contention or object-store latency, so they did not show the delays that people see. Per-endpoint Server-Timing only. It does not cover the many other requests that run during a task open. **Roadmap alignment** This is measurement for existing task-loading behaviour. It does not add product features. The staging baseline and the measured causes are in `doc/plans/2026-11-09-task-loading-performance.md`. Follow-up pull requests change the loading behaviour itself, starting with #20277. ## What Changed - Added `server/src/middleware/api-server-timing.ts`. Every `/api` response now has a `paperclip_api;dur=<ms>` Server-Timing entry. The value is the server time from request arrival until headers are written, including admission waits. The entry is appended, so the existing `paperclip_issue` phases stay first. - Registered the middleware first in `server/src/app.ts`. - Added `tests/perf/issue-detail/staging-click-harness.js`. It clicks a task row and records, frame by frame, when the composer and the latest message become visible, layout shifts, movement of the latest message, and the requests that run before the page is ready. The summary labels each sample instead of using the task reference. It replaces query values and every path segment that contains a digit. A fetch that is still running when the sample ends is listed as pending. - Documented the harness in `tests/perf/issue-detail/README.md`. - Added the dated plan `doc/plans/2026-11-09-task-loading-performance.md` with the staging baseline, the causes, and the prioritized slices. ## Verification - `cd server && npx vitest run src/__tests__/api-server-timing.test.ts`. 3 tests pass. They cover the JSON responses, the time spent in earlier middleware, preserving route phases, and explicit `writeHead` calls. - `tests/perf/issue-detail/issue-detail.perf.spec.ts` reads the first `dur=` value. That value is still `paperclip_issue`. - The harness was used on staging to capture the baseline in the plan document. ## Risks - Low risk. The header has only a duration, with no request, user or company data. - The middleware wraps `res.writeHead` once for each request. This is the same pattern that `on-headers` uses. - Server-Timing is visible to same-origin scripts only. No `Timing-Allow-Origin` header is added. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code with tool use (shell, browser automation, file editing). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |