Commit Graph
458 Commits
Author SHA1 Message Date
651749bdb4 feat(connections): AWS connector with credentials minted per connection (#22195)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents use outside services through connections, and the control
plane holds each connection's credentials
> - Companies want agents to "do things on AWS" with exactly the
permissions they choose, without Paperclip storing long-lived AWS keys
or rebuilding the AWS API
> - AWS hosts an MCP server that runs any AWS API call in its own
sandbox, and it accepts a one-hour bearer token that can be minted from
IAM credentials
> - This pull request adds an AWS connection that mints that token for
each connection at call time. The role method proves the connection with
the company's workload identity issuer (#22083), so no AWS key is stored
> - The benefit is that each AWS connection carries its own role and
permissions, AWS enforces them, and agents never see any AWS credential

## Linked Issues or Issue Description

Refs #22083 (workload identity issuer) and #22191 (spike findings).

**Subsystem affected**
Connections (tool access), tool gateway, shared app definitions,
instance settings.

**Problem or motivation**
Paperclip has no way to connect AWS. AWS's hosted MCP server needs a
bearer token from AWS Sign-in, minted from IAM credentials. No existing
credential path mints a header at call time from a role or key.

**Proposed solution**
Methods can declare `credentialMinter: "aws_mcp"`. The gateway and
connection health checks mint the `Authorization` header at call time,
and they share one in-memory cache per connection and grant (about 55
minutes). There are two methods:
- Role (`AssumeRoleWithWebIdentity` with a Paperclip-signed connection
token).
- Access key (the key directly, `GetFederationToken`, or `AssumeRole`).

Read-only connections add `ReadOnlyAccess` and
`AWSMCPSignInOAuthAccessPolicy` as session policies, so AWS refuses
changes. The feature is behind the `enableAwsConnector` experimental
setting.

**Alternatives considered**
- Run AWS's `mcp-proxy-for-aws` process on the control plane. Rejected:
it adds a Python runtime and processes, and gives nothing the token
exchange does not.
- Put credentials in the agent sandbox and use the AWS CLI. Deferred:
there is no per-call governance, and every sandbox would need the CLI.
- Add a new `credentialSource`. Rejected: it needs a migration, and the
`vercel_connect` database rules do not fit stored access keys.

## What Changed

- `packages/shared`:
- `credentialMinter` on connection methods. The validator requires
`mcp_remote` and `api_key`, and does not allow `keyPlacement` or another
credential source.
- The `aws` app definition (two methods, reviewed tools,
`mcpSessionRequired`, experimental key).
- The `enableAwsConnector` setting in the catalog, types and validators.
  - `aws` added to the connectable apps.
- `server/src/services/connectors/aws/mcp-credentials.ts`:
- Settings parsing and checks. The URL must be AWS's endpoint for the
saved region, and the role ARN must be valid.
- The minter: role or key session, read-only session policies, Sign-in
exchange, cache keyed on settings, key references and status, in-flight
de-duplication, eviction.
- A plain-language error mapping that never shows AWS's own text,
because that text can contain account IDs and ARNs.
- Gateway (`resolveCredentialHeadersUnrecorded`), credential access
(`resolveCredentialHeaders`), catalog discovery and health-failure
mapping:
  - The AWS branch.
- After a 401, the gateway evicts the token and mints again, then
requires an explicit retry (no replay). Discovery lists again.
- Setup:
- Minted methods may save a draft without contacting the provider. The
role must trust a connection that already exists.
- Saving an AWS role draft publishes the company's issuer, so AWS can
validate it when the customer registers it.
- `workload-identity.ts`: a draft connection can get a token, so setup
can test it. Disabled, archived or switched-off connections still get
none.
- `server/package.json`: `@aws-sdk/client-sts` and
`@aws-sdk/client-signin`. The lockfile is left to the refresh workflow.
- Artwork: AWS light and dark marks from LobeHub's MIT icon set (already
licensed in the repo). The manifest entry passes the brand checks.
- `scripts/ingest-app-definitions.mjs`: minted methods are exempt from
the `keyPlacement` requirement. The override source is
`scripts/app-definition-overrides/aws.json`. Permission reviews are
added.
- UI: the experimental settings toggle, in alphabetical position.
- Docs: `doc/connections/AWS.md`, the connections index, the issuer API
doc (draft rule), and the plan's phase table.

## Verification

- `server/src/services/connectors/aws/mcp-credentials.test.ts` (7 tests)
covers these items:
  - Both methods and the read-only policies.
  - Caching, revision, forced refresh, expiry and eviction.
  - Concurrent de-duplication.
  - The error mapping, with no AWS text leaking.
- Settings validation, including the URL and region check and the rule
that an unknown access level means read only.
- `server/src/__tests__/aws-mcp-connector.test.ts` (3 tests, real setup,
catalog and gateway code with a fake AWS and a session-requiring fake
MCP server) covers these items:
  - The setting gates setup.
  - The access key is never sent as a header or stored in config.
  - A cached token is reused.
  - After a 401, a fresh token is minted and the call is not replayed.
  - Role draft-then-resume publishes the issuer.
  - The signed token has the right `iss`, `sub` and `aud`.
  - An AWS refusal shows a plain health message without the account ID.
- Red/green:
  - Removing the gateway hook fails the access-key test.
  - Turning off `mcpSessionRequired` fails both integration tests.
- Removing draft support for minted methods fails both integration
tests.
  - Restoring each change passes again.
- Nearby suites: `tool-gateway-service`, `tool-access-service`,
`remote-mcp-connectors`, `instance-settings-service`,
`workload-identity`, plus the new files. 775 of 775 pass.
- Shared `app-definitions.test.ts` passes (42), including new
`credentialMinter` schema tests. UI `InstanceExperimentalSettings` (63)
and the brand asset tests pass. `check-app-brand-assets` validates 112
identities.
- Typecheck is clean for `packages/shared`, `server` and `ui`. The only
UI errors are in Storybook's generated catalog files, which this
worktree has not built.
- **Real AWS** (Paperclip dev account, integrated code path, issuer
behind a temporary public tunnel):
- Access key, read only: 8 tools listed. `run_script` `ListBuckets`
succeeded in 8.9 s on a cold sandbox. AWS refused `CreateBucket` ("no
session policy allows").
- Role: the draft was saved without contacting AWS, and the issuer was
published. After an IAM provider and a role trusting only that
connection were created, resuming setup listed 8 tools. `ListBuckets`
succeeded in 12.8 s. The role's own permissions refused `CreateBucket`.
- This run found two bugs the fakes did not: AWS needs MCP sessions for
calls, and the role method needs draft-first setup with the issuer
published. Both are fixed and both now have tests.
- All IAM users, keys, roles, providers and buckets were deleted, and a
listing confirmed it.
- Not run: Paperclip Cloud staging (the setting is off by default), and
`AssumeRole` from an access key with a role ARN against real AWS (unit
tests only).

## Risks

- The feature is off by default, behind `enableAwsConnector`. No
migration.
- New dependencies `@aws-sdk/client-sts` and `@aws-sdk/client-signin`
are in the existing `@aws-sdk` family (the server already uses
`client-s3`).
- Minted tokens are held only in process memory. A restart mints again.
A cached token can outlive a deactivated key by up to an hour, unless
AWS returns 401, which evicts it.
- `aws___run_script` is classified as a write, so the normal approval
and ask-first policy applies. Read-only connections rely on AWS session
policies.
- The guided setup screen and the CloudFormation template are not in
this PR. Setup uses the generic form and the documented draft and resume
steps.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, AWS SDK for JavaScript against the Paperclip dev AWS
account, a Cloudflare quick tunnel, an exploration subagent).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-18 12:48:49 -06:00
e2dda16b52 docs(plans): record the AWS connector spike findings (#22191)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The planned AWS connector gives each connection its own AWS identity
with no stored keys, using the workload identity issuer from #22083
> - The plan had open questions that only a real AWS account could
answer
> - This pull request records the answers from a spike against the
Paperclip dev AWS account
> - The benefit is that the connector build starts from proven facts,
and two design details are corrected

## Linked Issues or Issue Description

Refs #22083.

**Subsystem affected**
Connections (planned AWS connector), docs only.

**Problem or motivation**
The AWS plan assumed an `aws___call_aws` tool and a read-only session
policy with only `ReadOnlyAccess`. Neither had been tested.

**Proposed solution**
Record the spike results and correct the plan. The hosted server in the
test account offers `aws___run_script` but no `aws___call_aws`. A
read-only session also needs `AWSMCPSignInOAuthAccessPolicy`, or the
token exchange fails.

**Alternatives considered**
Leave the plan unchanged and find the differences during the build.
Rejected: the tool choice changes the governance design.

## What Changed

- `doc/plans/2026-11-17-aws-connector-and-minted-credentials.md`: a new
"Spike findings" section with results and timings, and corrections in
the minting, agent-call and governance sections. Phase 0 is marked done.

## Verification

- Spike against the dev AWS account. The issuer code from #22083 ran
locally behind a temporary public HTTPS tunnel. The spike verified these
items:
  - AWS accepted a Paperclip-signed token.
  - A token for another connection was denied.
  - The bearer exchange worked.
  - `tools/list` worked.
  - A read and a write through `run_script` worked.
  - A read-only session could read, and AWS refused its write.
  - Both access-key paths worked.
- After the spike, a listing of roles, OIDC providers, users and buckets
with the spike prefix returned nothing.
- Docs only. No code changed.

## Risks

- Low risk. Docs only.
- The tool list was observed in one account on one day. AWS can change
the hosted server.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, AWS SDK for JavaScript, AWS CLI, a Cloudflare quick tunnel).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-18 10:34:28 -06:00
DottaandClaude Fable 5.2 0acac00a49 docs(connections): record the staging tester's move to the separate Google app (#22187)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The built-in Google Workspace connector is going through Google
approval
> - `doc/memory/google-connector-memory.md` holds the decisions and
environment state for that work
> - The staging tester now uses the separate Google app, but the doc
still says it does not
> - This pull request records the switch, what was verified after it,
and what is still to confirm
> - The benefit is that the next session starts from the true state and
does not break staging deploys

## Linked Issues or Issue Description

Docs are out of date after the tester moved to the separate Google app.
Refs #22179.

## What Changed

- `doc/memory/google-connector-memory.md`: the tester is on the separate
Google app; what was verified; the consent-screen check that remains;
the deploy-order warning.
- `doc/plans/2026-11-17-google-workspace-production-checklist.md`:
ticked "wired to the new Google app".

## Verification

- Docs only. No code or tests changed.
- Checked that the diff has no client ids, project ids, stack ids,
passwords, or keys.

## Risks

- Low. Docs only. The doc is temporary and is deleted when the connector
is released.

## Model Used

- Anthropic Claude Fable 5.2 (`claude-fable-5-2`), in Claude Code with
tool use and browser control.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-18 16:20:32 +00:00
DottaandClaude Fable 5.2 b26c0edf23 docs(connections): record the staging Google tester state and findings (#22179)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The built-in Google Workspace connector is going through Google
approval
> - `doc/memory/google-connector-memory.md` holds the decisions and
environment state for that work
> - The staging tester now exists and passed its first live tests, but
the doc still says "to be created"
> - This pull request records the tester state, the remaining wiring
order, and the test findings
> - The benefit is that the next session starts from the true state and
does not break staging deploys

## Linked Issues or Issue Description

Docs are out of date. The working memory and the production checklist
say the staging tester does not exist. Refs #22050, #22065.

## What Changed

- `doc/memory/google-connector-memory.md`: tester state, what passed,
what is left, and four findings from live tests.
- `doc/plans/2026-11-17-google-workspace-production-checklist.md`: split
the tester item into "created" (done) and "wired to the new Google app"
(open).

## Verification

- Docs only. No code or tests changed.
- Checked that the diff has no client ids, project ids, stack ids,
passwords, or keys.

## Risks

- Low. Docs only. The doc is temporary and is deleted when the connector
is released.

## Model Used

- Anthropic Claude Fable 5.2 (`claude-fable-5-2`), in Claude Code with
tool use and browser control.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-18 15:37:40 +00:00
DottaandClaude Opus 5.5 f1abee7060 docs(plans): record staging results and plan the production rollout (#22181)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Paperclip Cloud now sends analytics directly to PostHog, and this is
live on staging
> - The plan doc records the rollout state and the steps that remain
> - Production needs an ordered, reversible rollout with clear decisions
> - This pull request records the staging results and adds the
production rollout plan
> - The benefit is one reviewed place that says what is done, what is
next, and how to roll back

## Linked Issues or Issue Description

**What is the problem?**
The plan doc still shows the staging rollout as pending and has no
production steps.

**What do you propose?**
Update the delivery state with the merged pull requests and staging
results, and add Phase 5: the production rollout, with prerequisites,
one step per switch, checks, and rollbacks.

**Why does it matter?**
Production changes must be ordered and reversible, and the open
decisions must be visible.

## What Changed

- Update the delivery state table in
`doc/plans/2026-11-17-cloud-direct-posthog.md`.
- Add "Phase 5: production rollout".

## Verification

- Documentation only.

## Risks

- Low risk. Documentation only.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 09:19:23 -06:00
11a0ff0205 feat(connections): per-company workload identity issuer with public signing keys (#22083)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents use outside services through connections. The control plane
holds each connection's credentials and makes the calls.
> - Some clouds, such as AWS, do not need stored keys. They trust a
signed token from an identity provider that they can verify against its
public keys.
> - Paperclip has no signing key whose public half it can publish. Agent
identity keys are Ed25519, which AWS does not accept, and the agent's
run receives the private key. Agent run tokens use a shared secret.
> - This pull request gives each company its own signing key and
publishes the public half as a standard OIDC issuer.
> - The benefit is that a connection, such as the planned AWS connector,
can get its own short-lived cloud credentials with no stored cloud keys.
The connection's role in the customer's account sets its permissions.

## Linked Issues or Issue Description

Refs #19414 and #10411. Those pull requests add OIDC *sign-in* to
Paperclip. This pull request is the other direction: Paperclip *issues*
tokens that outside clouds verify.

**Subsystem affected**
Connections (tool access) and the database.

**Problem or motivation**
An AWS connector should not store a long-lived AWS access key. AWS
supports OIDC federation (`AssumeRoleWithWebIdentity`). AWS fetches the
issuer's public keys over HTTPS and checks each token. Paperclip has no
key that it can publish for this, so this method is not available.

**Proposed solution**
Each company gets an ES256 key that Paperclip creates on first use. The
vault master key encrypts the private half. The issuer is
`https://{instance}/oidc/companies/{companyId}`. Paperclip publishes a
discovery document and a JWKS there. Tokens last 5 minutes and have `sub
= connection:{id}`. Only the control plane signs them. The board can see
key state and run planned rotation or emergency rotation. The design is
in `doc/plans/2026-11-17-workload-identity-issuer.md`. The AWS connector
plan that uses it is in
`doc/plans/2026-11-17-aws-connector-and-minted-credentials.md`.

**Alternatives considered**
- Reuse agent identity keys. Rejected: AWS does not accept Ed25519, and
the agent's run receives the private key.
- Derive the key from the master key, with no table. Rejected: the
master key alone would then sign as production. Worktree environments
copy the master key on purpose. A master key rotation would also change
the signing key and break every customer trust.
- Use one issuer for the instance. Rejected: if a customer's role checks
the issuer but not `sub`, any company on a shared instance could assume
that role.

## What Changed

- `packages/db`: new table `company_workload_identity_keys` and
migration `0498`. The migration only creates a new table, so the
previous release keeps working. Partial unique indexes allow one
`active` key and one `next` key per company.
- `server/src/services/workload-identity.ts`: lazy key creation with
concurrent convergence, a private-key cache with a fail-closed key match
check, the discovery document, the JWKS, `signConnectionToken`, and
rotation steps. A key that is no longer active stays published for 24
hours. Paperclip calculates this when it reads the keys, so no
background job is necessary.
- `server/src/routes/workload-identity.ts`: public `GET
/oidc/companies/{companyId}/.well-known/openid-configuration` and `GET
/oidc/companies/{companyId}/jwks.json` (404 until a key exists). Board
`GET /api/companies/{companyId}/workload-identity` and `POST
.../workload-identity/rotation`, with an activity log entry for each
rotation.
- Token issuance writes a tool-access audit event. The event never
contains the token.
- `cli`: worktree seeds exclude the new table in the minimal mode and
the full mode. This is the same as for `agent_identity_keys`, so a
development copy never holds production's signing key.
- Docs: `docs/api/workload-identity.md`, an entry in the API overview,
and the two plan docs.
- No connector uses `signConnectionToken` yet. The AWS connector pull
request will use it.

## Verification

- `cd server && npx vitest run src/__tests__/workload-identity.test.ts`:
8 tests pass. The tests cover these items:
  - The issuer requires https with no port and no loopback host.
  - The endpoints return 404 before a key exists.
  - Six concurrent first uses create one key.
  - The JWKS contains only `kty crv x y kid alg use`.
- A token verifies against the published key and has the expected
claims.
- A company cannot sign for another company's connection, and its keys
do not verify that company's tokens.
  - Signing stops when the public address is not usable or has changed.
  - Two-step rotation and emergency rotation work.
- Board-only access is enforced, and each rotation writes an activity
log entry.
- Red/green: I put back a JWK that leaks `d` and removed the company
check on the connection. Three tests failed. I restored the code, and
all 8 tests passed.
- `cd cli && npx vitest run src/__tests__/worktree.test.ts`: 63 tests
pass.
- `npx tsc --noEmit` passes for `packages/db`, `server` and `cli`.
- I started this branch on a real dev server. Before a key existed, the
public routes returned JSON 404 (not the UI page). After an emergency
rotation from the board route, the JWKS returned one public P-256 key.
On the http dev instance, discovery stayed 404 and the board route
reported `not_https`.
- `pnpm verify` exited 2 because of the migration. The required CI gates
supply the broad coverage.
- Not run: a real AWS `AssumeRoleWithWebIdentity` call. That needs a
public HTTPS instance. It is the first step of the AWS connector work
and will run against staging.

## Risks

- The migration is low risk. It creates a new table with no backfill,
and the compatibility checker classifies it as compatible with the
running release.
- The public routes are unauthenticated. They show only public keys,
return 404 for companies without a key, and use random company UUIDs.
Each request is one small indexed read.
- The issuer URL shape is permanent after customers trust it. A change
to the instance's public address breaks those trusts until the customer
updates the identity provider. `signConnectionToken` stops on that
mismatch when the caller supplies the recorded issuer.
- Anyone who holds the production database and the master key could sign
for that instance's companies. That is the same exposure as every other
vault secret.
- Edge routing on Paperclip Cloud must pass `/oidc/...` to the stack. I
will confirm this on staging with the AWS connector work.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, web fetch of AWS documentation, a local dev server).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-17 23:33:47 -06:00
43b2d8be22 feat(dashboard): add an experimental areas and responsibilities dashboard (#22069)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The dashboard is the first page a person sees in a company
> - The classic dashboard shows agents, runs, costs and recent tasks
> - It does not show what the person is responsible for, or if that work
goes well
> - People think in areas, such as Fundraising or Customers, and in the
things they own in each area
> - This pull request adds a dashboard built on areas and
responsibilities, behind an experimental setting that is off by default
> - The benefit is that a person sees what they own, how it is going,
and what agents do for it, on one page

## Linked Issues or Issue Description

No public issue exists for this work. The description follows the
feature request template.

**Subsystem affected**

Dashboard, navigation, the task conversation, routines, and the instance
experimental settings.

**Problem or motivation**

The classic dashboard counts agents and tasks. It does not say what a
person owns or if it goes well. People who run a company with agents
want to see their areas, the responsibilities in each area, the bar each
one must meet, and the agents and routines that serve it.

**Proposed solution**

A person owns responsibilities. Each responsibility belongs to an area
and has a bar, a number to reach. Projects, tasks and routines link to a
responsibility, and the page shows its status from those links. An agent
can interview the person and draft the dashboard. The person accepts or
declines each change, and any change can be undone from a saved version.

**Alternatives considered**

Add the same data to the classic dashboard. That page is a fixed set of
cards for company activity, so a second, person-centred page is clearer.
The setting lets a person pick one of the two.

**Roadmap alignment**

This is an experimental feature behind `enableResponsibilities`. It is
off by default and changes nothing for a person who leaves it off.

## What Changed

- Add the setting "Areas & Responsibilities dashboard"
(`enableResponsibilities`, off by default). On, Dashboard in the sidebar
opens the new page, and the classic dashboard link is not shown in the
sidebars or the command palette. Off, nothing changes.
- Add tables and migrations 0492 to 0497 for areas, responsibilities,
links to work, metrics, status reports, saved dashboard versions, and
model-call records for summaries.
- Add the API: areas, responsibilities, links, bars, metrics, status
reports, summaries, runs, automation, and the dashboard conversation
with versions and undo. Each route checks company access and writes to
the activity log.
- Add four task conversations with an agent: about a bar, about an
automation, about a routine, and about the whole dashboard. The agent
proposes a change. Nothing applies until a person accepts it.
- Add standing checks: routines that an agent carries out on its
heartbeat and reports.
- Add the Dashboard page: search, counts, areas that open and close, a
primary New area button, and a message box at the bottom of the page.
Add the responsibility page with its bar, metrics, work and routines.
- Add the Paperclip skill sections and runner API entries that let an
agent use these routes. The runner skill inventory is regenerated (166
to 174 rows).
- Keep the new runner API tests in their own file so
`runner-api.test.ts` stays at 985 lines.
- Onboarding is not changed. The onboarding changes that build on this
work are a separate pull request.

## Verification

- Type checks pass for `packages/db`, `packages/shared`, `server` (all
layers) and `ui`.
- `pnpm --filter @paperclipai/db check:migrations` passes. A new
`drizzle-kit generate` run produces no further change.
- Tests for every changed file pass: 26 server test files (3,009 tests),
116 UI test files (1,991 tests) and 28 shared test files (432 tests).
- The runner capability inventory checks pass.
- I started the app on a local database that already held earlier data.
Master's new migrations and these migrations applied without error. With
the setting on, the Dashboard and a responsibility page loaded with no
console or server errors. The More menu and the command palette show one
Dashboard.
- I did not run `pnpm verify`, the full local suite, or the browser
suites. Required CI covers them.
- The agent conversations are tested with simulated agents. No real
model has run them.

## Risks

- Medium. The change is large, and the setting is off by default. With
it off, the new routes refuse requests and the UI is unchanged.
- Migrations only add tables and columns. Two replace a CHECK
constraint, and one builds an index `CONCURRENTLY`. These two have the
`online_compatible` override with a reason in the file, and the second
one is listed in `MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS`.
- Master can add migrations before this merges. The migrations then need
new numbers.
- `responsibilities.ts` (1,663 lines), `responsibility-summaries.ts`
(1,239 lines) and one Storybook fixture file (1,305 lines) are new files
over 1,000 lines. They each keep one cohesive unit together: the service
for one table family, the model-call path for summaries, and the
fixtures that the stories share. No existing file crosses 1,000 lines.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude: Claude Fable 5.2 (`claude-fable-5-2`) wrote most of
the change, with Claude Sonnet 5.5 (`claude-sonnet-5-5`) for the final
steps. Both used tool use, code execution and browser checks.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Scott Tong <scott@scottsmbpm5max.lan>
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-17 20:45:44 -08:00
DottaandClaude Opus 5.5 56023fe80f feat(google-workspace): gateway sets the REST tool result size limit for all six services; update connector docs (#22050)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents read Gmail, Calendar, Drive, Docs, Sheets and Slides through
the built-in Google Workspace connection
> - Paperclip now has its own REST-backed Google tools in
`server/src/google-workspace-rest-tools/`. No service uses them yet
> - That folder guessed a result size limit of 250,000 bytes. It applied
the limit only to Drive, Docs, Sheets and Slides. Gmail and Calendar had
no limit
> - The guess was wrong in both directions. It refused documents that
Paperclip can deliver, and the folder should not know Paperclip's limits
> - This pull request makes the limit a number that the gateway gives.
The folder applies it to every tool in all six services through one
check. It also brings three Google planning docs up to date
> - The benefit is that a REST result is refused only when the gateway
would also refuse the same result from Google's own MCP server

## Linked Issues or Issue Description

Refs #22027, #22028, #22029, #22026

**Is your feature request related to a problem?**
The REST tools refused results over a size the folder chose itself
(250,000 bytes). Paperclip can deliver larger results: the runner saves
a large result as an artifact and gives the agent a pointer. Gmail and
Calendar results had no limit at all.

**Describe the solution you'd like**
The gateway gives the limit. The folder applies the given number to the
final serialized result of every tool, with one helper. A result over
the limit is an error that says how to ask for less. It is never cut
short.

**Describe alternatives you've considered**
A larger number chosen from the runner's 1 GiB capture limit. Google's
own transfer limit (20 MB per response) and memory use make that a poor
choice. Also, results would then differ between the REST path and
Google's MCP path.

### Limits a tool result passes on the way to an agent

| Point | Limit | What happens over it |
| --- | --- | --- |
| Google response read by `google-http.ts` | 20 MB
(`MAX_RESPONSE_BYTES`) | Tool error. Nothing returned |
| REST folder result (new) | the number the gateway passes: 999,885
bytes | Tool error with "read less" advice. The gateway returns
`tool_error` (502) |
| Gateway read of a remote MCP response (`MAX_REMOTE_MCP_RESPONSE_BYTES`
in `tool-gateway.ts`) | 1,000,000 bytes per JSON body or event-stream
message | `mcp_remote_response_too_large` (502). This is the "1,000,000
bytes are rejected" statement in `doc/prompt-injection-prevention.md`.
It applies to the Google MCP path only; the REST path has no HTTP read |
| Injection screening model check | 256 KiB model request | Not a
delivery limit. A larger result is checked by the built-in patterns only
|
| Gateway HTTP route (`POST /tool-gateway/.../tools/call`) | none in
Paperclip | `res.json(result)` |
| Runner (`captureRunnerToolResult`) | 500,000 bytes
(`RUNNER_TOOL_RESULT_MAX_BYTES`) | Saved as an artifact. The agent gets
a pointer and reads it with `read_tool_result` |
| Runner capture | 1 GiB (`RUNNER_API_RESPONSE_MAX_BYTES`) |
`SemanticToolOutcomeUnknownError` |

The gateway wraps the result as `{ content, data: { content,
structuredContent, ... } }`, so the payload occurs again. Measured: a
Gmail result of 676,459 bytes from the folder became 1,017,386 bytes
delivered (ratio 1.50). The ratio cannot be more than 2.

### The budget

`GOOGLE_WORKSPACE_REST_MAX_RESULT_BYTES` in `tool-gateway.ts` =
`MAX_REMOTE_MCP_RESPONSE_BYTES` less the JSON-RPC envelope and
event-stream framing (`event: message\ndata:
{"jsonrpc":"2.0","id":"paperclip-tool-<uuid>-version","result":...}`) =
**999,885 bytes**. A result that Google's MCP server could deliver
through the gateway is also delivered over REST, and the reverse. With
the wrapper, the largest delivered result is less than 2 MB. The runner
saves it as an artifact (far below 1 GiB). So the folder never refuses a
result that the runner can give as a pointer.

Practical limits at that budget (plain ASCII; escaped and non-ASCII
characters count more):

| Tool | Largest content |
| --- | --- |
| Gmail `get_message` | about 500 KB plain body, or about 250 KB each
for plain plus HTML |
| Calendar `get_event` | about 500 KB description |
| Drive `read_file_content` | about 500 KB text |
| Drive `download_file_content` | about 375 KB file bytes |
| Docs `read_doc` | about 240 KB document text |
| Slides `read_presentation` | about 250 KB text |
| Sheets `get_values` | about 500 KB cell text, at most 50,000 cells |

## What Changed

- `tool.ts`: `defineTool` takes `whenTooLarge` advice per read tool
(required for reads, refused for writes) and a `ToolLimits` argument. It
measures the final `CallToolResult` once (`withinResultBudget`) and
replaces an oversized result with a `too_large` tool error.
`FALLBACK_MAX_RESULT_BYTES` (250,000) applies only when a direct caller
gives no limit.
- `callGoogleWorkspaceRestTool` and `createGoogleWorkspaceRestServer`
take `maxResultBytes`. A value that is not a positive integer is refused
before any request.
- All 23 read tools state their advice: a smaller `pageSize` for list
tools, `messageFormat` or `view` for Gmail, a narrower range for Sheets,
`fields` for Slides. Write tools are not checked: a completed write is
never reported as an error, so an agent is not led to repeat it.
- Drive, Docs, Slides and Sheets no longer have their own byte checks
(`boundedStructuredResult`, `boundedResult`, `sizedResult`,
`MAX_CONTENT_BYTES` removed). Drive downloads stop reading at the given
limit. The Sheets 50,000-cell limit stays.
- Two Drive tool descriptions no longer state a fixed number.
- Adapter `google-workspace-rest-dispatch.ts`: `maxResultBytes` is a
required input and is passed through.
- `tool-gateway.ts`: new `GOOGLE_WORKSPACE_REST_MAX_RESULT_BYTES`,
passed on the REST branch.
- Folder README: new "Result size" section with the limits and costs;
old 250,000-byte text replaced.
- Docs: `doc/memory/google-connector-memory.md`, the production
checklist and the code-fix plan now list #22026 to #22029, the REST
status (36 of 36, none switched on), the November 17 decisions, the
known differences from Google's tools, open risks and next steps.

## Verification

- `cd server && npx vitest run src/google-workspace-rest-tools` passes
(53 files, 1,361 tests with the adapter test).
- New `result-budget.test.ts`: for Gmail (read, list), Calendar (list,
read), Drive, Docs, Slides and Sheets, a result just under a given
budget is returned and one just over is refused with the tool's advice.
Without a budget, the fallback applies; with a larger budget, the same
result is returned. Drive downloads use the given cap. A Gmail list
tells the caller to lower `pageSize`. A Gmail draft result over the
budget is still returned, not an error. `defineTool` refuses a read tool
without advice and a write tool with it.
- `npx vitest run src/services/google-workspace-rest-dispatch.test.ts`:
the adapter passes the gateway's number, and the folder applies exactly
that number.
- `npx vitest run src/__tests__/google-workspace-rest-transport.test.ts`
(24 tests): with Gmail on REST, a 330 KB email passes the gateway with
screening, is more than 500,000 bytes delivered, and
`captureRunnerToolResult` turns it into a `read_tool_result` pointer. A
520 KB email is refused with `tool_error` and the advice, after one
Google request.
- `npx tsc --noEmit -p server/tsconfig.json`: no errors.
- `pnpm verify --explain`: broader verification required (a doc under
`doc/` is a global CI input). CI supplies that coverage.
- Time per call: the check is one `JSON.stringify` of the result. About
5 µs for a 4 KB result, 18 µs for 42 KB, 0.24 ms for 700 KB. Drive,
Docs, Sheets and Slides had the same pass before, so they add nothing.
Gmail and Calendar add this one pass.

## Risks

- Gmail and Calendar results now have a limit. Direct callers that pass
no limit get the 250,000-byte fallback. The gateway always passes its
number.
- Drive, Docs, Slides and Sheets accept results up to about 4 times
larger than before through the gateway. Large results reach agents as
artifact pointers.
- Error text for oversized results changed to one shared form.
- No service is switched to REST, so users see no change today.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code with
tool use (shell, file edits, GitHub CLI) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 13:05:13 -06:00
DottaandClaude Opus 5.5 7f1f0eef67 fix(google-workspace): request free/busy with read & write Calendar, and update connector docs (#22026)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The built-in Google Workspace connector gives agents Gmail, Drive
and Calendar through one connection, behind an experimental setting
> - The user picks each service as read only or read & write. Paperclip
then requests only scopes from a fixed list of 14
> - Read & write Calendar requested `calendar.calendarlist.readonly` and
`calendar.events`. Our code assumed `calendar.events` also covers
free/busy lookups
> - Google's REST `freeBusy.query` does not accept `calendar.events`. So
the REST `suggest_time` tool cannot read free/busy on a read & write
grant, and falls back to reading each attendee's events
> - This pull request adds `calendar.events.freebusy` (already one of
the 14) to the read & write Calendar selection of the unified
connection, and stops treating `calendar.events` as implying free/busy
> - The benefit is that `suggest_time` on REST can see coworkers who
share only free/busy, and the stored grant no longer claims a permission
that Google did not give. It also brings three Google connector planning
docs up to date

## Linked Issues or Issue Description

Refs #22016, #22017, #22010, #22014

**What happened?**

The read & write Calendar selection of the unified Google Workspace
connection requests `calendar.calendarlist.readonly` and
`calendar.events`. `toolScopes.calendar.suggest_time` lists
`calendar.events` as sufficient, and `googleWorkspaceEffectiveScopes`
treats `calendar.events` as implying `calendar.events.freebusy`.

Google's two references differ:

- REST
[`freeBusy.query`](https://developers.google.com/workspace/calendar/api/v3/reference/freebusy/query),
checked 2026-11-17: "This request allows authorization with at least one
of the following scopes:" `calendar.readonly`, `calendar`,
`calendar.events.freebusy`, `calendar.freebusy`. `calendar.events` is
not in the list.
- Google's hosted Calendar MCP
[`suggest_time`](https://developers.google.com/workspace/calendar/api/v3/reference/mcp/tools_list/suggest_time),
checked 2026-11-17: "Requires one of the following OAuth scopes:"
`calendar`, `calendar.readonly`, `calendar.events`,
`calendar.events.freebusy`, `calendar.freebusy`. Here `calendar.events`
is accepted.

So `calendar.events` is enough for Google's MCP tool but not for the
REST endpoint that our REST `suggest_time` uses.

**Expected behavior**

A user who picks read & write Calendar gets a grant that allows
`freeBusy.query`. A grant that does not have the free/busy permission is
not recorded as having it. Grants made before this change keep working
as read & write.

**Steps to reproduce**

1. Call `googleWorkspaceSelectedScopes({ calendar: "write" })`. It
returns two scopes, without `calendar.events.freebusy`.
2. Call `googleWorkspaceEffectiveScopes(readOnlyCalendarScopes,
[calendar.events])`. It returns `calendar.events.freebusy`, which Google
did not grant.
3. With Calendar on the REST transport, `suggest_time` on a read & write
grant gets an insufficient-scope answer from `freeBusy.query` and uses
the events fallback.

## What Changed

- `packages/shared/src/google-workspace-connectors.ts`:
- `googleWorkspaceSelectedScopes` adds selection-level scopes from a new
`SELECTION_ADDED_SCOPES` map. Read & write Calendar adds
`calendar.events.freebusy`. The per-service `calendar.write` profile
(`SERVICE_PROFILES["calendar.write"]`) is unchanged, because the Cloud
broker matches legacy profiles by exact scope set.
- `googleWorkspaceEffectiveScopes` no longer expands `calendar.events`
to `calendar.events.freebusy`. `calendar.events` still implies
`calendar.events.readonly`. `calendar.readonly` still implies free/busy,
which Google's reference allows.
- `googleWorkspaceSelectionForScopes` is unchanged. It needs the legacy
write profile's scopes, so a two-scope grant and a three-scope grant
both read back as read & write.
- `toolScopes.calendar.suggest_time` is unchanged. It still accepts
`calendar.events`: Google's MCP tool accepts it, and on REST the tool
has an events-based fallback that names attendees it cannot read.
- Counts: default all-read-and-write selection is now 10 scopes (was 9);
all read-only stays 8; the union of all 27 selections stays exactly the
14.
- Tests (written first and seen failing on `master`):
- `packages/shared/src/google-workspace-connectors.test.ts`: scopes for
read and for read & write Calendar; the legacy `calendar.write` profile
keeps two scopes; default count 10 and read-only count 8; old two-scope
grant, new three-scope grant and partial consent (user declines
free/busy) all round-trip as read & write and keep `list_calendars`,
`list_events`, `suggest_time`, `create_event` and `delete_event`; no
recorded free/busy that Google did not grant. The existing exhaustive
test still checks that no selection requests a scope outside the 14.
- `server/src/__tests__/tool-access-service.test.ts`: customer-owned
Workspace OAuth with read & write Calendar asks Google for all three
scopes. With and without free/busy in Google's answer, the connection is
active, the Calendar tools are listed, the grant stores exactly what
Google granted, and it reads back as read & write.
- Stale tests on `master` fixed: `#22014` made Gmail 9 of 9 in the REST
tool set while `#22016` still expected 3 Gmail REST tools.
`google-workspace-rest-dispatch.test.ts` and
`google-workspace-rest-transport.test.ts` now derive the Gmail list from
the allowlist and use Chat as the service with no REST tools.
- Docs: `doc/connections/GOOGLE-WORKSPACE.md`,
`doc/connections/GOOGLE-WORKSPACE-SERVICE-CONSENT.md` and
`doc/connections/tool-method-permission-reviews.json` say ten scopes and
explain the Calendar change.
- Docs: `doc/memory/google-connector-memory.md` (shipped PRs #22010,
#22014, #22016, #22017; three Nov 17 decisions; "Where Google content is
kept" replaced with #22010's inventory, what deletion keeps and its
limits; Scopes section; REST tools status table; still-open list),
`doc/plans/2026-11-17-google-workspace-production-checklist.md` (help
page and retention audit ticked with #22010; REST tool progress),
`doc/plans/2026-11-17-google-workspace-code-fixes.md` (fix 6 done with
limits; section 9 status table; fix 10 note).
- `tests/runner-e2e/source-sdk-public-inputs.json`: refreshed pins for
the changed shared files.

Not changed:

- `server/src/google-workspace-rest-tools/**`. Another change owns that
folder. Its README open question about this scope is now resolved and
can be removed there.
- The picker disclosure (`googleWorkspaceDisclosure.ts`). Read & write
Calendar already says agents "can see your calendars, events and
free/busy times". With this change that line is accurate on REST too.

Behavior for existing grants: a saved read & write grant without
free/busy still reads back as read & write, keeps every Calendar tool,
and refreshes without change. Nothing prompts the user to reconnect. A
user who chooses read & write Calendar in the picker is asked for the
new scope. An account-page reconnect without a new choice still restores
the exact saved request, as before.

Cloud broker and Google Console: no change needed.
`calendar.events.freebusy` is one of the 14 in `workspace.all`, the
broker accepts any nonempty subset of `workspace.all`
(`doc/connections/GOOGLE-WORKSPACE-SERVICE-CONSENT.md`), and the legacy
`calendar.write` profile's exact scope set did not change.

## Verification

- `cd packages/shared && npx vitest run
src/google-workspace-connectors.test.ts src/app-definitions.test.ts`: 55
passed. On `master` the new shared tests fail (3 failed).
- `cd server && npx vitest run src/__tests__/tool-access-service.test.ts
-t "free/busy"`: 2 passed. With `master`'s
`google-workspace-connectors.ts`: 2 failed.
- `cd server && npx vitest run
src/__tests__/google-workspace-rest-transport.test.ts
src/services/google-workspace-rest-dispatch.test.ts`: 29 passed (4
failed on `master` before this change).
- `cd ui && npx vitest run
src/features/connections/GoogleWorkspaceServices.test.tsx
src/pages/apps/AppDetail.test.tsx`: 115 passed.
- `cd packages/shared && npx tsc --noEmit -p .`: clean.
- `pnpm verify` exits 2 (a global CI input changed), so required CI
supplies the broad coverage.
- Docs facts checked against `gh pr view` for #22010, #22014, #22016,
#22017, the code, and paperclipai/paperclip-website PR 197 (still a
draft; does not mention OpenRouter).

## Risks

- Low risk. The submitted set of 14 does not change, and the broker
contract does not change.
- New read & write Calendar grants record `calendar.events.freebusy`
next to `calendar.events`. Code that reads stored scopes sees one more
scope. All consumers (callback, refresh, discovery, picker pre-fill) are
covered by the tests above.
- A read-only Calendar request on an account that Google reports only
`calendar.events` for no longer records free/busy. That matches what
Google granted; `suggest_time` is then not offered on that grant.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use, web fetch and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 16:40:58 +00:00
DottaandClaude Opus 5.5 1ca90d2d6f docs(connections): record Nov 17 Google Workspace progress in memory, checklist and plan (#22007)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The built-in Google Workspace connector is going through Google's
restricted-scope review
> - Three docs track that work: the working memory, the production
checklist and the code-fix plan
> - Four pull requests merged on November 17 and one more is open, so
the docs are out of date
> - This pull request updates the three docs to match the merged work
and the Cloud audit results
> - The benefit is one current place for decisions, scopes, shipped work
and open questions

## Linked Issues or Issue Description

Refs #21996, #21998, #21999, #22000, #22001

**Issue type**
Outdated documentation.

**Where is the issue?**
`doc/memory/google-connector-memory.md`,
`doc/plans/2026-11-17-google-workspace-production-checklist.md` and
`doc/plans/2026-11-17-google-workspace-code-fixes.md`.

**What's wrong?**
The docs describe fixes 1 to 4 as not done. They plan a new npm package
for the REST tool set, but the tool set is a folder inside the server.
The scope table is in two files. The docs do not record the Cloud broker
audit or where Google content is kept.

**Suggested fix**
Record what shipped, add the new decisions, keep one scope table, add
the broker and data-retention findings (marked as reported), and tick
the checklist items that are now true.

## What Changed

- Working memory: three new decisions (no new npm packages, the scope
handling contract, agents merge their PRs on the coordinator's approval
at green checks and Greptile 5/5).
- Working memory: new "What has shipped" table (#21996, #21998, #21999,
#22001 and #22000, with merged commits).
- Working memory: the single copy of the 14-scope table. "Known
mismatches" now lists only what is still true after #22001.
- Working memory: new "Cloud broker (paperclip-cloud)" and "Where Google
content is kept" sections, from the November 17 read-only audit, marked
as reported and not re-verified.
- Working memory: how to create the staging tester, its two obstacles,
and two new open decisions with a recommendation.
- Working memory: the privacy policy draft location and what it must
also cover (the OpenRouter screening model and the picker's retention
statement).
- Working memory: a note that the ponytail reviewer flagged the summary
of Google's reply. The summary stays.
- Checklist: the scope table is now one sentence with a link. Four
product items are ticked with date and PR link. Items that are not fully
true stay unticked.
- Code-fix plan: fixes 1 to 4 marked done with PR links. Section 9
layout is now the folder inside the server. Two one-line lessons.
Section 10 lists the broker changes and the deploy order.

## Verification

- Each fact was checked against `gh pr view` for #21996, #21998, #21999,
#22000 and #22001, and against
`packages/shared/src/google-workspace-connectors.ts` and
`server/src/services/tool-access/oauth/grant-refresh.ts` on
`origin/master`.
- The broker and data-retention facts come from an audit that this PR
does not re-verify. The docs say so.
- Searched the changed files for client IDs, secrets and project
numbers: none.
- Docs only. No tests apply.

## Risks

- Low risk. Docs only. No code or behavior changes.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, file edits, GitHub CLI).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 08:40:25 -06:00
DottaandClaude Opus 5.5 3293700448 docs(plans): Cloud sends analytics directly to PostHog (#22003)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Paperclip Cloud is the hosted version, and it shares the product
telemetry path with self-hosted installs
> - Self-hosted telemetry goes through the Paperclip collector, which
anonymizes every identifier
> - Cloud needs real identifiers, page analytics, and session replay in
PostHog to find and fix bugs for specific users
> - This pull request adds the plan that moves Cloud to direct PostHog
delivery and keeps the self-hosted path unchanged
> - The benefit is one reviewed plan that the follow-up pull requests
implement

## Linked Issues or Issue Description

**What is the problem?**
The PostHog Product analytics and Web analytics pages show no data for
Paperclip Cloud. Only server events arrive, and they carry anonymized
install identifiers.

**What do you propose?**
Cloud sends events directly to PostHog with real identifiers.
Self-hosted installs keep the collector path. Existing Cloud stacks keep
their PostHog history through a precomputed legacy identifier.

**Why does it matter?**
The team can trace a bug from a specific user or workspace to events and
session replays.

## What Changed

- Add `doc/plans/2026-11-17-cloud-direct-posthog.md`.

## Verification

- Documentation only. No code changes.

## Risks

- Low risk. Documentation only.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 08:38:49 -06:00
DottaandClaude Opus 5.5 12cb18e17d docs(connections): Google Workspace approval checklist, code-fix plan and working memory (#21996)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The built-in Google Workspace connector lets agents read and write
Gmail, Drive, Docs, Sheets, Slides and Calendar data
> - The connector stays behind an experimental setting until Google
approves its data access and the security assessment is accepted
> - The approval work has many steps, and the decisions behind them were
spread across chats and private notes
> - This pull request adds one checklist, one code-fix plan and one
temporary working-memory file for that effort
> - The benefit is that contributors and agents share one source for the
scope contract, the decisions and the remaining work

## Linked Issues or Issue Description

**Type of issue**
Missing documentation for an in-progress effort.

**Description**
The Google Workspace connector needs Google approval before it can go to
all users. There is no single document that lists what "approved and
released" means, which code changes must land before submission, or the
decisions taken so far (launch transport, owner, test instance, 14-scope
contract).

**Proposed change**
Add a production and approval checklist, a code-fix plan for section 2
of that checklist, and a temporary working-memory file. Keep
identifiers, credentials and review correspondence out of the public
repository.

## What Changed

- Added `doc/plans/2026-11-17-google-workspace-production-checklist.md`:
the "done means" criteria, the 14-scope contract, and checklists for
decisions, baseline, product and code, policy, security assessment,
reviewer environment, live acceptance, demo video, submission, release
and ongoing work.
- Added `doc/plans/2026-11-17-google-workspace-code-fixes.md`: ten
ordered code fixes (disclosure in the picker, scope lock to 14,
`gmail.modify` removal from the managed client, injection screening
check, revoke on last disconnect, retention audit, scan findings, gate
removal, REST-backed tool set, scope cleanup). Each item is marked
confirmed or to check.
- Added `doc/memory/google-connector-memory.md`: a temporary file with
decisions, open questions, environments, Google app layout, scopes,
review history and privacy-policy gaps. It says to delete it once the
connector is released.

## Verification

- Docs only. No code, schema or test changes.
- Read all three files before the push and confirmed that they contain
no passwords, client IDs, client secrets, project numbers or review
login codes.
- Relative links between the three files and to
`doc/connections/GOOGLE-WORKSPACE.md` resolve in the repository.
- CI checks on this PR.

## Risks

- Low risk. Documentation only.
- The plans describe intended work. Some items are marked "to check" and
can change after the code is traced.
- The working-memory file is temporary and must be deleted after
release.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), used through Claude
Code with tool use (shell, git, GitHub CLI) to commit the files and open
this PR.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 06:57:30 -06:00
DottaandClaude Opus 5.5 3706f520fd test(scaling): keep ramping through recovered runs, read spend from cost records, and wait for termination before deleting agents (#21776)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The scaling effort uses a real-agent ramp harness
(`scripts/scale-agents-ramp.ts`, #21695 and #21757) to load a hosted
test instance through the product API
> - Its first run with agents that really ran (10 agents, 105 tasks, 106
runs) stopped on one failed run, recorded $0 model spend, and left
cleanup errors
> - Read-only inspection of the instance showed three harness faults.
The failed run had a same-run bootstrap retry scheduled 33 s later, but
the harness stopped and cleaned up first. Run rows carry no cost, but
cost records do (63.7 cents for the run's agents). Terminated agents
leave the company agent list, so the harness treated them as done and
sent deletes that Paperclip refused with 409
> - These faults make the ramp stop too early, make its spend cap inert,
and make cleanup report the wrong cause
> - This pull request keeps the ramp going through runs that Paperclip
recovers by itself, reads spend from cost records, records board latency
per route, and makes cleanup wait for termination and name what it left
behind
> - The benefit is that the next ramp grades the product, not the
harness, and its spend cap works

## Linked Issues or Issue Description

Refs #21695
Refs #21757

**What happened?**
The ramp stopped at step 1 with `failed_runs_zero failed (observed 1)`.
The run failed with `runner_ingress_bind_conflict` before its provider
session opened. Its run events show `same-run resume persisted` with
`recoverable: true` and `nextAttemptAt` 33 s later. The harness stopped
within 9 s and paused the agents, so the retry was blocked
(`native_retry_admission_ineligible`). The result showed `Recorded model
spend: $0`. The harness read `costUsd` from run rows, which is always
null; `GET /api/companies/<id>/costs/by-agent` holds 63.71 cents over
104 cost events for the 10 agents. Cleanup reported two `DELETE
/api/agents/:id returned 409: Complete termination before deleting the
agent`. Those agents were still `lifecycleState: terminating`; the
harness waited on the company agent list, which hides terminated agents,
so it treated them as done. A later `--cleanup-only` changed nothing
because it also found agents only through that list. The board API p90
failed (765 ms), but the result had no per-route timings to say which
request was slow.

**Expected behavior**
A failed run whose task finishes by itself is counted, not a stop. A
failed run whose task does not finish within a grace stops the ramp. The
share of failed runs is graded at the step end. Spend comes from
Paperclip's cost records. Cleanup deletes an agent only after
termination completes, and names the agents and sandbox leases still
pending.

**Steps to reproduce**
1. Run `node --test scripts/scale-ramp/scale-ramp.test.ts` with the new
fake and tests and the `origin/master` `ramp.ts`, `cleanup.ts`,
`criteria.ts` and `report.ts`.
2. 16 tests fail, among them every new test (recovered run, unrecovered
run, spend from cost records, spend cap, board per route, termination
wait, still-terminating agents and `--cleanup-only`, leases released
during the wait).
3. With this change, all 60 pass.

**Paperclip version or commit**
Instance on `2026.1106.0+2564.git.3244780bd`; harness from #21757.

## What Changed

- `criteria.ts`: replace `failed_runs_zero` with
`failed_runs_unrecovered_zero` (immediate) and `failed_run_rate` (at the
step end). Step metrics add `runsEnded`, `runsFailedRecovered`,
`runsFailedUnrecovered`, `runsFailedPending` and `board.byRoute`.
- `ramp.ts`: a run counts as failed once it was seen `failed` or
`timed_out`. It is recovered when its task reaches `done`. It is
unrecovered when the task rests not done, needs Retry, or has no live
run for `--recovery-grace-minutes` after the failure. A task whose
interruption has an automatic retry scheduled (`state: retry_scheduled`,
`nextRetryAt` or `retryWait`) does not count as needing Retry. Spend
comes from `costs/by-agent` for the run's agents on each poll, and each
step reports the spend in its window. Board probes keep a distribution
per route.
- `config.ts`: add `--max-failed-run-percent` (default 2) and
`--recovery-grace-minutes` (default 15).
- `cleanup.ts`: read agents by id as well as from the list, so
terminated agents in the state file are found. Wait for `lifecycleState`
`terminated` or `rejected` by id before a delete. Report
`agentsStillTerminating`, `activeSandboxLeaseIds` and
`sandboxLeasesReleasedDuringWait`, and name each lease still active with
its run and acquire time.
- `report.ts`: the summary shows recovered, unrecovered and pending
failed runs, board latency per route, and each cleanup error.
- `fake-paperclip.ts`: run rows carry no cost; `costs/by-agent` returns
cents per agent. Optional failure of the n-th run, a scheduled recovery
run, a termination delay (terminated agents leave the list and a delete
answers 409 until termination completes), and leases that stay active
after their run.
- `doc/plans/2026-11-13-thousand-concurrent-agents.md`: the runbook
describes the failed-run criteria, the spend source and the cleanup
behavior.

## Verification

- `node --test scripts/scale-ramp/scale-ramp.test.ts`: 60 of 60 pass.
- Red check: with the new fake and tests but the `origin/master`
`ramp.ts`, `cleanup.ts`, `criteria.ts` and `report.ts`, 16 tests fail,
including all 8 new ones.
- `tsc -p scripts/scale-ramp/tsconfig.json`: no errors.
- Evidence from read-only GETs on the test instance: run `f1892d43`
events (bind conflict, scheduled same-run retry, blocked after the
pause), run rows (`costUsd` null on all 106), `costs/by-agent` (63.71
cents for the 10 agents), agent rows by id (`terminating` while absent
from the list), lease rows.
- Not run: a new real ramp (needs the owner's go-ahead).

## Risks

- Low risk. The change touches only the scale-ramp harness, its fake,
its tests and one runbook section. No product code changes.
- A failed run no longer stops the ramp at once. An unrecovered one
stops it after the grace (15 min by default), so a broken step can run
up to that long. The step-end failed-run rate still fails a step with
more than 2% failed runs.
- Spend now needs cost read access for the board token. If
`costs/by-agent` cannot be read, the result reports spend as not
measured and the spend cap does not trigger.
- Cleanup can now take up to 5 more minutes while it waits for
termination by id.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, read-only HTTP GETs to the test instance).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 09:39:28 -06:00
DottaandClaude Opus 5.5 15afc32137 test(scaling): wait for agent setup and gate the ramp on a first started run (#21757)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The scaling effort uses a real-agent ramp harness
(`scripts/scale-agents-ramp.ts`, added in #21695) to load a hosted test
instance through the product API
> - Its first real run held a 20-minute step with 10 tasks, 10 queued
runs and 0 started runs, and reported only "concurrency_reached failed
(observed 0)"
> - The cause was in the harness: it bound the agents to a plugin
Runner-task environment, which runs only agents with a managed AI
connection. Each agent stayed in setup (status `paused`, setup state
`verifying`) with that reason, so each run stayed queued. The product
did what it is designed to do
> - The harness did not wait for agent setup, did not read the setup
error, and held the full step with nothing running. Its fake API created
agents ready at once, so its tests could not catch this
> - This pull request makes the harness wait for agent setup, gate the
ramp on one started run, and stop within minutes with Paperclip's own
reason
> - The benefit is that a wrong environment or any other start blocker
costs about one minute and prints the cause, not a 20-minute step with
no explanation

## Linked Issues or Issue Description

Refs #21695

**What happened?**
A ramp of 10 then 50 agents on a hosted test stack created 10 agents, 10
tasks and 10 runs. In 20 minutes no run started. The result said only
"concurrency_reached failed (observed 0)". Cleanup found every agent
already paused and cancelled 10 queued runs. The tasks never left `todo`
(status version 1, no checkout run). The agents were bound to the
"Paperclip Sandbox (Daytona)" plugin environment, whose `daytona-tasks`
driver declares `supportsTasks: true`. For such an environment,
lifecycle verification refuses a `paperclip_runner` agent that has no
`runtimeConfig.aiConnection` (`ai_connection_required`, see
`server/src/services/agent-harness-verification.ts` and its test
"refuses a plugin Runner-task agent without a managed AI connection
before allocating a target"). The failure pauses automatic verification,
so the agents never become ready.

**Expected behavior**
The harness must not create a full step of work before agents can run.
When agents cannot finish setup, or when no run starts, it must stop
soon, print the task, run and agent state with Paperclip's reason, and
clean up.

**Steps to reproduce**
1. Run `node --test scripts/scale-ramp/scale-ramp.test.ts` with the new
fake (agents start in setup; `agentSetupError` keeps them there) and the
old `ramp.ts`, `setup.ts` and `config.ts`.
2. The new test "stops within minutes with Paperclip's reason when the
agents never finish setup" fails with `concurrency_reached`, the same
result as the real run.
3. With this change, it stops with `agents_ready` and the setup reason
in under 5 virtual minutes.

**Paperclip version or commit**
Instance on `2026.1106.0+2555.git.819f61f1f` (master at 11:34Z on
2026-11-15); harness from #21695.

## What Changed

- `setup.ts`: add `waitForAgentsReady`, `agentReady` and
`describeAgent`. The harness reads the company agent list until each
agent has setup state `ready` and is not paused. It stops when an agent
reports a setup error for 60 s, or after `--agent-ready-minutes`
(default 10), and prints the agent's status, setup state, holds and
Paperclip's reason.
- `ramp.ts`: before the first step, a preflight creates one agent and
one task and creates the rest only after that task's run starts. A new
check stops the ramp when no run has started within
`--start-timeout-minutes` (default 5) of the first task. The stop reason
names the task status, any execution blocker, the run status, error
code, error and liveness, and the agent's setup state. New stop
criteria: `agents_ready` and `run_started`. Run records keep `error`,
`errorCode` and `livenessReason`.
- `config.ts`: add `--start-timeout-minutes` and
`--agent-ready-minutes`.
- `fake-paperclip.ts`: agents start as the real product creates them
(status `paused`, setup state `preparing`) and become `ready` after
`agentReadyDelayMs`, or stay in `verifying` with `agentSetupError`.
Issue creation queues a run even for an agent in setup, as the real
product does. Runs start only for a ready agent.
- `scale-ramp.test.ts`: three tests. Agents that never finish setup stop
the run with `agents_ready` and the reason, with one agent and no task
created. A first task that never starts stops the run at the preflight
with `run_started`, the task, run and agent state, and one cancelled
run. The rest of a step is created only after the preflight task starts.
- `doc/plans/2026-11-13-thousand-concurrent-agents.md`: the Phase 3
runbook describes the start checks, the first run's cause, and which
environments the harness can use.

## Verification

- `node --test scripts/scale-ramp/scale-ramp.test.ts`: 43 of 43 pass.
- Red check: with the new fake and tests but the `origin/master`
`ramp.ts`, `setup.ts` and `config.ts`, 3 tests fail. The setup test
fails with `concurrency_reached` (the real run's result) where
`agents_ready` is expected.
- `tsc -p scripts/scale-ramp/tsconfig.json` (same options): no errors.
- Read-only inspection of the hosted instance (GET only): task rows
(`todo`, status version 1, no checkout run), activity (agent created, no
pause, 10 `heartbeat.cancelled` at cleanup), the plugin manifest
(`daytona-tasks` `supportsTasks: true`), and the company AI connections
(no OpenRouter connection). The run and agent rows were deleted with the
agents at cleanup, so their setup error was inferred from the code path
and the cleanup counts, not read.
- Not run: a new real ramp (needs the owner's go-ahead).

## Risks

- Low risk. The change touches only the scale-ramp harness, its fake and
its tests, and one runbook section. No product code changes.
- A real ramp now spends up to one preflight start (seconds to minutes)
before the first step. The preflight task is part of step 1 and the
preflight running samples are dropped, so step metrics keep their
meaning.
- An agent whose setup error clears within 60 s does not stop the run.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, read-only HTTP GETs to the test instance).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 08:07:03 -06:00
DottaandClaude Opus 5.5 b5ba548112 feat(server): add an instance-admin server metrics endpoint for the scale ramp (#21708)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The scaling plan
(`doc/plans/2026-11-13-thousand-concurrent-agents.md`) has a finish line
that a real-agent ramp harness checks (#21695)
> - Four finish-line criteria need numbers from inside each server
process: main-thread CPU, run event lateness, agent API time before the
handler, and missed lease renewals
> - The server had no way to give them, so every harness run was at best
"incomplete"
> - This pull request adds one instance-admin endpoint that returns
those numbers per process over a short sliding window, off by default,
and connects the harness to it
> - The benefit is that a ramp run can measure every server criterion it
can see, with near-zero cost when on and no cost when off

## Linked Issues or Issue Description

Refs #21695

**Subsystem affected**

Server (instance routes, run event append, native lease renewal),
database client (statement counter), and the scale-ramp harness in
`scripts/scale-ramp/`.

**Problem or motivation**

The scale-ramp harness reads `PAPERCLIP_SCALE_METRICS_URL`, but the
server has no endpoint for it. The criteria `main_thread_cpu`,
`run_event_lateness_p99`, `agent_api_pre_handler_p99` and
`lease_renewals_missed_zero` are therefore always "not measured", and no
run can pass.

**Proposed solution**

Add `GET /api/instance/server-metrics`. It returns a JSON snapshot of
one server process over the last 50 to 60 s. Only an instance admin can
read it. The env flag `PAPERCLIP_SERVER_METRICS` (read once at start,
default off) turns on collection and the route. Off, the path answers
404 and no collector starts.

**Alternatives considered**

- The postgres.js `debug` option for the statement counter: rejected,
because it captures a stack (`new Error()`) for every query.
- Process CPU from `process.cpuUsage()` as `mainThreadCpu`: rejected,
because it includes GC and libuv threads. It is returned as `processCpu`
instead, and `mainThreadCpu` reads the main thread from `/proc` (Linux
only, null elsewhere).
- Default on: the measured cost is negligible (numbers below), but the
event-loop monitor wakes the process every 20 ms when idle, and this is
a new instance-wide surface. It stays off; the runbook says how to turn
it on for a ramp run.

## What Changed

- `server/src/services/server-metrics.ts`: a per-process collector. It
uses `performance.eventLoopUtilization()`, `monitorEventLoopDelay` (20
ms), `process.cpuUsage()`, the main thread's
`/proc/self/task/<pid>/stat` on Linux, and six 10 s buckets of native
`perf_hooks` histograms. One unref'd timer rotates the buckets. Nothing
on the hot path awaits, allocates per call or does I/O.
- `server/src/routes/instance-server-metrics.ts`: `GET
/api/instance/server-metrics`, guarded by the existing
`assertInstanceAdmin` (the check that `/instance/database-backups` and
other instance routes use). It returns numbers, a random per-process
`processId`, `uptimeSec`, `buildRevision` and `windowMs`. It returns no
SQL, names, IDs or hostnames.
- `server/src/app.ts`: when the flag is on, two middlewares time agent
API requests from arrival to the API router, the route is mounted, and
shutdown stops the collector. When the flag is off, nothing is mounted.
- The snapshot also carries `dedicatedConnections` from
`dedicatedDbConnectionStats()` (#21693): `limit`, `inUse`, `waiting` at
request time and `peakInUse`, `queued`, `overLimit`, `nestedOverLimit`
since start. The harness ignores it today; it shows dedicated-connection
queueing in a restart burst.
- `packages/db/src/statement-count.ts` and
`lazy-transaction-queries.ts`: a Drizzle logger that increments one
counter per statement replaces Drizzle's no-op logger. No edit to
`client.ts`.
- `paperclip-control-plane-port.ts`: one call records runner `emittedAt`
to committed row for each committed (not duplicate) runner event.
- `native-execution-lease.ts`: one call counts a lease whose renewal
deadline passed.
- Harness: missed lease renewals are summed per `processId` (several
processes, or a restart, no longer cancel out). A metrics URL on the
target origin gets the board token in the `Authorization` header;
`PAPERCLIP_SCALE_METRICS_TOKEN` still overrides it, and another origin
gets no token.
- Contract: `scripts/scale-ramp/server-metrics.fixture.json` is the
serializer's own output. A server test asserts the serializer produces
exactly this file, and a harness test parses it and checks every field.
- `rules/company-scope-exceptions.mdc`: adds the snapshot as an
instance-wide exception with its reason. The runbook in the plan doc
documents the path, the flag, the credential and each field.

| Field | Provided | Source |
| --- | --- | --- |
| `mainThreadCpu` | yes (Linux; null elsewhere) |
`/proc/self/task/<pid>/stat` utime+stime |
| `eventLoopUtilization` | yes | `performance.eventLoopUtilization()` |
| `eventLoopDelayP99Ms` | yes | `monitorEventLoopDelay`, highest 10 s
p99 in the window |
| `statementsPerSecond` | yes | Drizzle logger counter (not BEGIN/COMMIT
or raw driver calls) |
| `poolWaitP99Ms` | absent | postgres.js has no cheap hook for queue
time |
| `runEventLatenessP99Ms` | yes (null with no events) | runner event
append |
| `agentApiPreHandlerP99Ms` | yes (null with no agent requests) |
arrival and router-entry middlewares |
| `leaseRenewalsMissed` | yes, counter since process start | native
lease deadline timer |

## Verification

- `cd server && npx vitest run
src/__tests__/instance-server-metrics-routes.test.ts
src/__tests__/server-metrics.test.ts`: through `createApp` on the
embedded test database. 200 with the exact field set for an instance
admin; 403 for a company owner who is not an instance admin, for an
agent key, and unauthenticated; 404 and no collector when the flag is
off; values move after agent requests, statements, a late run event and
a missed renewal; the body has no IDs, keys or SQL.
- `cd server && npx vitest run
src/services/native-runtime/native-execution-lease.test.ts
src/services/native-runtime/paperclip-control-plane-port.test.ts
src/__tests__/api-server-timing.test.ts`: the lease and run-event hooks
(red/green checked for the lease hook by removing the call).
- `cd packages/db && npx vitest run src/lazy-transaction-queries.test.ts
src/client-options.test.ts`: the counter counts statements inside
transactions and savepoints.
- `node --test scripts/scale-ramp/scale-ramp.test.ts` and `tsc -p
scripts/scale-ramp/tsconfig.json`: 38 pass, including the fixture
contract and the board-token header.
- `node server/scripts/typecheck-layers.mjs` and `tsc --noEmit` in
`packages/db`: pass.

Overhead, measured on a shared, loaded laptop (load average 8 to 20),
median with min and max:

| Measure | Off | On |
| --- | --- | --- |
| Logger call per statement (5M calls x 7) | 0.28 ns (0.24 to 0.97) |
3.36 ns (3.32 to 6.16) |
| CPU per `select 1` statement, embedded Postgres (3,000 x 10) | 28.7 us
(20.3 to 42.4) | 32.3 us (20.7 to 45.5) |
| CPU per HTTP request, client and server in one process (2,000 x 6) |
67.0 us (40.4 to 107.4) | 64.2 us (40.9 to 83.3) |
| Idle process CPU, 5 s x 3 | 0.01% of a core (0.00 to 2.21) | 0.08%
(0.05 to 0.09) |
| Record one run event (Date.parse + histogram) | - | 148 ns (135 to
188) |
| One snapshot | - | 55 us |

The statement and request differences are inside the run-to-run spread.
The direct cost is about 3 ns per statement, against about 30 us of CPU
per statement.

## Risks

- Low risk. Off by default; off, only the statement counter runs, and it
costs one increment more than Drizzle's no-op logger.
- `mainThreadCpu` assumes Linux USER_HZ of 100 and is null on other
platforms. The harness then uses `eventLoopUtilization`.
- Run event lateness compares the runner's clock with the server's
clock, so clock skew adds to it. Negative values count as zero.
- With several processes, each request reaches one process. The harness
keeps the highest gauge and sums counter growth per process, but a
process it never samples is not seen.
- The statement counter does not count BEGIN/COMMIT or raw postgres.js
calls.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, file edits, local test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 21:13:45 -06:00
DottaandClaude Opus 5.5 328a954cc1 test(scaling): add a real-agent ramp harness for hosted instances (#21695)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The control plane must run 1,000 agents at once on one workspace
(`doc/plans/2026-11-13-thousand-concurrent-agents.md`)
> - Above about 50 agents, the evidence is a load model only. Nobody has
run 200 or 1,000 real agents
> - Phase 3 of the plan proves the finish line with real runs on hosted
instances, in a fixed order
> - This pull request adds the harness for those runs. It does not run
it against any real instance
> - The benefit is that each proof step is one command with a written
cost estimate, explicit pass/fail criteria, and a guaranteed cleanup

## Linked Issues or Issue Description

**What problem does this solve?**
The 1,000-agent plan needs real runs at 50, 200, 500 and 1,000 agents.
No tool exists that creates real agents and tasks on a hosted instance,
ramps concurrency, measures the control plane, and judges the plan's
finish line.

**Proposed solution**
A script under `scripts/` that ramps real agents through the product
API, stops at the first failed criterion, writes a machine-readable
result, and always cleans up what it created.

**Alternatives considered**
Extend the restart-chaos hosted mode. That mode grades release swaps at
up to 30 concurrent tasks. The ramp reuses its target guard and its
cleanup order, but it has a different workload, different measures and a
different stop rule.

## What Changed

- `scripts/scale-agents-ramp.ts` and `scripts/scale-ramp/`: the harness.
It reads the base URL from a flag or the environment and the credentials
from the environment only. It creates or reuses a company, then creates
agents, tasks, a project and an optional provider secret. It ramps
concurrency in steps (default 10, 50, 100, 200, 500, 1,000) and keeps
each agent on one small deterministic task at a time.
- It measures per step: tasks started, finished, failed and needing
Retry; failed runs; peak running agents; queue wait; time to first token
and first activity (from sampled run events); run duration; board GET
latency. An optional metrics endpoint supplies event-loop, statement and
pool figures. A missing source is reported as "not measured", never as
zero.
- It evaluates the criteria from the plan's finish line plus harness
criteria, writes them into `result.json` and `SUMMARY.md`, and stops at
the first failure.
- Cleanup runs in a `finally`: pause agents, cancel live runs, archive
tasks, terminate and delete agents, delete the stored secret, archive
the project and the created company, and wait for sandbox leases.
`--cleanup-only <run id>` finishes a cleanup after a crash. `--dry-run`
prints the plan and the estimate and makes no request.
- Target guard: `--target-kind` is required and must match the host.
Shared staging and production hosts need a second explicit flag. The
script asks for the host name before it starts unless `--yes` is given.
A run above the spend threshold needs `--confirm-spend <max USD>`.
- `scripts/scale-ramp/scale-ramp.test.ts`: 28 tests against an
in-process fake API with a virtual clock. They cover ramp logic, stop on
failure, criteria evaluation, cleanup after a mid-run crash and an
interrupt, `--cleanup-only` with and without a state file, dry run, the
production-host guard, the spend gate, and that credentials never appear
in output or files.
- `scripts/scale-ramp/tsconfig.json` is added to the test programs that
CI typechecks. The test file joins the `test:release-registry` node test
list, which CI runs.
- `doc/plans/2026-11-13-thousand-concurrent-agents.md`: a Phase 3
runbook with commands, cost, criteria, proof order, target prerequisites
and open risks.

## Verification

- `node scripts/ci-profile/node-test.mjs
scripts/scale-ramp/scale-ramp.test.ts
scripts/typecheck-test-programs.test.mjs`: 47 tests pass.
- `node node_modules/typescript/bin/tsc -p
scripts/scale-ramp/tsconfig.json`: no errors.
- `node cli/node_modules/tsx/dist/cli.mjs scripts/scale-agents-ramp.ts
--base-url https://scale1.test.paperclip.app --target-kind test-stack
--steps 50 --hold-minutes 30 --dry-run` prints the plan, the estimate
and the criteria, and makes no request.
- The harness was not run against any real instance. That needs the
owner's go-ahead.

## Risks

- Low risk for the product: the change adds a script, tests and docs
only. No server code changes.
- A real run creates real agents, tasks and sandboxes and spends money.
The guard, the typed confirmation, the spend gate, the task cap and the
recorded-spend stop limit this. The runbook lists the open risks.
- The cost defaults are assumptions until a first real run records
actual spend.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 20:10:51 -06:00
DottaandClaude Opus 5.5 c66530da63 fix(server): release settled native run process identity at graceful shutdown (#21689)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A server process records the local process of each native run in
`heartbeat_runs` (`process_pid`, `process_group_id`,
`process_started_at`, `process_host_hash`)
> - Only a server in the same PID space can prove that such a process
stopped. `proveLocalOwnerGone` does not accept a recorded process group
from another container
> - Each deploy replaces the server process. Settled native runs that
the old server ran keep their process fields, and no later server can
release them. Recovery sweeps had to learn to skip these rows one by one
> - The stopping server is the one process that can check its own PID
space. After its graceful drain, it can see that a settled run's process
exited
> - This pull request adds a graceful-shutdown step that releases the
process identity of those runs, with the same rules that the running
server already uses, in one candidate read and one update
> - The benefit is that a deploy no longer adds stranded process records
for the runs that the stopping server could prove stopped, and a
successor on the same host has nothing to release at startup

## Linked Issues or Issue Description

Refs #21590

**What happened?**
After a graceful shutdown, settled native runs (`failed`, `cancelled`,
`timed_out`, `interrupted`) that the stopping server ran keep
`process_pid` and `process_group_id`. The successor runs in a new
container with a different `process_host_hash`. It cannot prove that the
recorded process stopped, so the rows stay forever. A successor on the
same host releases them at startup with about 10 statements for each
run.

**Expected behavior**
The server that ran the process knows that the process exited. A
graceful shutdown should release these records before the server stops.
An abrupt stop should keep today's behavior.

**Steps to reproduce**
1. Run native work so that some runs settle while the server runs, or
settle during the shutdown drain.
2. Send SIGTERM to the server.
3. Count settled native runs with `process_pid` or `process_group_id`
set: the count does not go down after the deploy.

**Paperclip version or commit**
master at cf2a4221e3

**Additional context**
This change does not clear historical rows. It does not change
`proveLocalOwnerGone`.

## What Changed

- `releaseOwnedSettledNativeRunProcessesAtShutdown`
(`native-local-process-stop.ts`): one read selects at most 500 settled
native runs whose `process_host_hash` is this server's PID space and
that pass the existing release rules (settled coordinator phase, a
checkpoint or provider evidence, no sandbox lease, no pending local
lease, a matching `native.process_identity_recorded` event from this
host). The server checks that each PID and process group is absent. One
statement in one transaction then clears the fields and appends the
`native.local_process_stopped` receipt with the original identity. The
update checks every rule and the exact recorded identity again, so a
changed row is not released.
- The transaction sends the same work signals as the per-run release
(execution replacement, live-run recovery, retained session cleanup,
accounting, waiting instruction copies). It sets a 2 s
`statement_timeout`.
- `notifyInstructionCopyStopEvidenceForRuns`
(`instruction-collection-notifications.ts`): the batch form of
`notifyInstructionCopyStopEvidence`, with the same waiting-copy
predicate.
- `server/src/index.ts`: a bounded shutdown step
`settled_native_process_release` (3 s) runs after the heartbeat drain
and the finalizer drain, before the database closes. A timeout or
failure does not stop the shutdown.
- `doc/plans/2026-11-13-thousand-concurrent-agents.md`: the measured
result under "Background recovery sweeps".

## Verification

- `pnpm exec vitest run
src/__tests__/shutdown-settled-process-release.test.ts` (embedded
PostgreSQL):
- 40 own settled runs with an exited process, plus 6 runs that must
stay: a running (restart-suspended) run, a settled run with a live
process, a sandbox run without a termination receipt, a run from another
container, a run with an unsettled coordinator, and a run with pending
lease cleanup. Before: 45 stranded settled runs. After: 5 (the 5 settled
runs that must stay). The release used 7 statements for 40 runs. The 6
kept runs keep their exact fields and get no receipt. Each released run
has one receipt, and `withReleasedNativeRunProcess` restores its
identity.
- Successor on the same host: `discoverSettledNativeRunProcesses` used 1
statement after a graceful shutdown and 401 statements (40 releases)
after an abrupt stop.
  - Abrupt stop: no step runs, and all rows and receipts stay as today.
- A run whose recorded identity changes between the read and the update
is not released.
- Red check: with the release step changed to do nothing, 3 of 4 tests
fail.
- Regression: `settled-native-process-rechecks.test.ts` and
`heartbeat-native-restart-suspension.test.ts` (127 tests),
`agent-instruction-copy-retry-backoff.test.ts` (12 tests),
`task-retry.test.ts` and `heartbeat-process-recovery.test.ts` pass. `tsc
--noEmit` for `server` passes.
- No server was started.

## Risks

- Low risk. The step releases only what the same server would release in
its next recovery pass, with the same rules, and only in its own PID
space. It does not touch running runs, so a restart-suspended native run
keeps everything that the resume path reads.
- The step is bounded (500 runs, 3 s, 2 s statement timeout). The
release is one transaction. A kill in the middle leaves the rows as they
were.
- Runs still running at shutdown, sandbox runs, coordinator `observed`
rows, controller leases and remote workspace writer markers do not
change. The open decision about the PID space of a stopped container
stays open.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 19:18:39 -06:00
DottaandClaude Opus 5.5 3eabdd6d10 perf(server): size run-log uploads and object journal IO from open run logs (#21685)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - One server process controls every run of a workspace, and the target
is 1,000 concurrent agents on one server
> - Each native run writes its run log to object storage. The server
batches a run's log lines and uploads one batch per run each second
> - The server allowed only 4 log uploads and 8 object storage
operations at a time, for the whole process. These numbers were set when
50 agents was a lot
> - At 200 runs, the run logs fell about 8 seconds behind. At 1,000
runs, pending log bytes grow toward the 64 MiB process cap. Past that
cap, run events wait and log appends can fail
> - This pull request sizes the log upload slots and the object storage
operation slots from the number of open run logs, between the old floor
and a ceiling that comes from the storage client's socket budget
> - The benefit is that run logs stay about one flush interval behind at
200 runs, and fall 7.7 seconds behind instead of 44 seconds at 1,000
runs (p50), without more sockets than the storage client allows

## Linked Issues or Issue Description

Refs plan item 1g in
`doc/plans/2026-11-13-thousand-concurrent-agents.md` ("Size the shared
limiters ... from the active run count").

**What existing behavior does this improve?**
The shared run-log upload limiter and the shared object journal
operation limiter.

**Subsystem affected**
Server: object storage run logs (`object-run-log-store.ts`,
`object-byte-journal.ts`) and the S3 storage client.

**Current behavior**
The process allows 4 run-log uploads and 8 object journal operations at
a time, for any number of runs. Each upload is 3 to 4 sequential object
writes. At 200 runs and 50 ms per write, the run logs commit with a p50
delay of 8.0 s, and only 55% of the log lines offered in a 20 s window
commit inside it.

**Proposed behavior**
The upload slots are one per 10 open run logs, at least 4 and at most
21. The operation slots are the upload slots plus 4 for reads, at most
25. That is half of the 50 sockets that the S3 client may open, so
downloads and other storage calls keep the other half. The S3 client now
sets its 50-socket limit explicitly, from the same constant.

**Reason and benefit**
With the change, 200 runs commit their logs at the flush interval (p50
1.1 s), and 1,000 runs commit with a p50 delay of 7.7 s instead of 43.8
s. Fewer pending bytes stay in memory, so the 64 MiB pending cap and its
admission failure are farther away.

**Breaking changes**
None. With 40 or fewer open run logs the limits are the same as before
(4 uploads, 8 operations).

## What Changed

- `server/src/lib/active-run-slots.ts`: a small helper that gives a slot
count from an active run count, clamped between a floor and a ceiling.
- `server/src/storage/s3-client-limits.ts` and `s3-provider.ts`: the S3
client sets `maxSockets` to 50 (the SDK default) from a named constant,
so the budget is explicit.
- `object-byte-journal.ts`: the object operation limit is the upload
slots plus a read reserve of 4, from the number of open run-log writers.
Open writers are counted with `trackObjectJournalWriter()`. When the
limit grows, waiting operations start at once. When it shrinks, running
operations finish and no new ones start until the count is below the
limit.
- `object-byte-journal.ts` also owns the upload limiter
(`acquireObjectJournalUpload`), process-wide, next to the writer count.
When a run log opens, waiting uploads start at once. When run logs
close, running uploads finish and no new upload starts until the running
count is below the new limit.
- `object-run-log-store.ts`: uses the shared upload limiter. A run log
registers when it begins and releases when it is finalized. The waiting
queue stays first in, first out, and each run has at most one upload in
flight, so one run cannot starve another.
- Review fix: the first revision kept the upload limiter per store, so a
run log opened by another store did not wake uploads waiting in this one
(a lost wakeup on growth). The shared limiter fixes it; a new test
covers growth and shrinkage with queued work.
- Tests: limit growth with uploads waiting and shrinkage with uploads
running; helper bounds; the storage socket budget; 200 open runs upload
at least 20 logs at once (4 before); 1,000 open runs stay at the ceiling
and every upload finishes.

## Verification

- `cd server && npx vitest run src/lib/active-run-slots.test.ts
src/services/object-run-log-store.test.ts
src/services/object-byte-journal.test.ts`: all pass.
- Red/green: with the slot count forced back to the fixed floor of 4,
the two new scale tests fail (4 uploads in flight); with the change they
pass.
- Simulation (in-memory object store with 50 ms per write, each run
appends 1.5 KB at 0.7 records/s, the recorded native event rate, 20 s
window, flush every 1 s). Counts are stable on a loaded machine because
the latency is simulated:

| Runs | Build | Peak concurrent writes | Writes/s | Log lines committed
in window / offered | Commit delay p50 / p99 |
| --- | --- | --- | --- | --- | --- |
| 200 | before | 4 | 74 | 1,473 / 2,669 | 8.0 s / 13.5 s |
| 200 | after | 20 | 211 | 2,646 / 2,790 | 1.1 s / 1.3 s |
| 1,000 | before | 4 | 75 | 1,229 / 13,499 | 43.8 s / 62.3 s |
| 1,000 | after | 21 | 392 | 7,789 / 13,667 | 7.7 s / 12.6 s |

## Risks

- More object storage writes per second at high run counts (at 1,000
runs and 50 ms per write, about 390/s instead of about 75/s). This
raises object storage request cost in proportion. The ceiling bounds it.
- The S3 client now states its socket limit. The value is the SDK
default, so behavior does not change.
- The open-writer count is process-wide. A run log that is never
finalized keeps its count, as it already keeps its writer entry. The
ceiling bounds the effect.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 18:52:23 -06:00
DottaandClaude Opus 5.5 e8c79235ca perf(attention): scope run lifecycle retention hints and read only used columns (#21579)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The `attention-retention` queue archives old attention items (failed
runs, reviews, questions, recovery actions, blocked tasks). A company
hint makes it reload every candidate of the company, 100 per page, with
eligibility reads for each page.
> - Several run lifecycle producers sent that company hint: every agent
status change after a run, the execution-control writes and legacy run
terminalization when they name only a run, scheduled retries, and the
Runner's task monitor tool.
> - Each page also read whole rows it did not use: full `heartbeat_runs`
rows (recent ones carry tens of KB of `context_snapshot` and
`runner_profile_json`), full `issues` rows (descriptions), and the whole
agent roster with its configuration.
> - This pull request names the task graph or the source kind in those
hints, and reads only the columns each eligibility decision uses.
> - The benefit is fewer company reloads per run event and far fewer
result bytes per reconstruction, with the same decisions.

## Linked Issues or Issue Description

No issue exists. This describes the problem with the fields of
`.github/ISSUE_TEMPLATE/enhancement.yml`. Stacked on #21552 (it builds
on that pull request's change to `execution-control-notifications.ts`);
review that one first.

**What existing behavior does this improve?**

The attention retention queue
(`server/src/services/attention-retention.ts`) and the producers that
wake it on run lifecycle paths.

**Subsystem affected**

Server background retention (`attention-retention.ts`,
`retention-work-notifications.ts`, `execution-control-notifications.ts`,
`heartbeat/agent-status.ts`, `heartbeat/retries.ts`,
`native-runtime/paperclip-runner-tool-authority.ts`).

**Current behavior**

- `heartbeat/agent-status.ts` sends `{ companyId }` on every agent
status change after a run (two places). `heartbeat/retries.ts` sends it
for each scheduled retry. `writeExecutionControl` and
`notifyExecutionRunWork` send it when the scope names only a run. The
Runner's `set_task_monitor` tool sends it for one task.
- On a local fixture with 1,500 open tasks and run churn, the queue
loaded 47 company candidate rows per second from these hints.
- `pageEvidence` read `select *` from `heartbeat_runs`,
`issue_recovery_actions` and (through `getTableColumns`) `issues`;
`reviewEligibility` read `select *` from `issues`;
`interactionEligibility` read the full agent roster; the per-entry
`check` repeated full-row reads.

**Proposed behavior**

- Agent status changes send `{ companyId, sourceKind:
"agent_error_alert" }`. The status moves between `running`, `idle` and
`error`. All three are invokable for blocker attention
(`BLOCKER_ATTENTION_INVOKABLE_AGENT_STATUSES`) and for questions
(`agent-eligibility.ts`). Failed-run attention only excludes terminated
agents, and this path never sets that. So only agent error alerts can
change.
- Run-scoped execution-control writes, legacy run terminalization and
scheduled retries use `notifyIssueAttentionRetentionWork` with the run
id (the run's task, its parents and the tasks it blocks; company
fallback past 100). This is the same helper as #21521. The task monitor
tool scopes to its task.
- `notifyIssueAttentionRetentionWork` matched the run's saved task id
with `issue.id::text = …`. It now validates and casts the id, so the
primary key is used, as #21552 does for recovery.
- Narrow reads, same decisions:
- runs: `id`, `agent_id`, `created_at` and the saved task id computed in
SQL with the same rules as `readRunIssueId` (`issueId` unless it is
absent or JSON null, then `taskId`; only non-empty strings).
  - recovery actions: `id`, `source_issue_id`.
- issues for evidence and `visibleIssue`: the blocker-attention input
columns plus `unblock_descriptor` and `blocked_transition_at`.
- review issues: `id`, `company_id`, `status`, `execution_state`,
`assignee_user_id`.
- agent roster for question addressees: `id`, `company_id`, `name`,
`reports_to`, `status` (the `AgentOrgRow` that
`evaluateAgentInvokability` takes).

**Reason and benefit**

A run event no longer reloads every attention candidate of the company.
A reconstruction no longer transfers whole run, task and agent rows.

## What Changed

- `server/src/services/attention-retention.ts`: column lists for runs,
recovery actions, issues and the roster; the run probe type replaces
whole rows.
- `server/src/services/heartbeat/agent-status.ts`,
`heartbeat/retries.ts`, `execution-control-notifications.ts`,
`native-runtime/paperclip-runner-tool-authority.ts`: scoped attention
hints.
- `server/src/services/retention-work-notifications.ts`: indexed lookup
of a run's saved task.
- `server/src/__tests__/attention-retention-work.test.ts`: two tests.

## Verification

- `reads only the fields its decisions use from large runs, reviews and
agents` (I saw it fail first): 50 failed runs, 50 tasks in review with
large descriptions, 10 questions to a paused agent, about 40 KB of
payload in each run's snapshot, profile and stdout and in the agents'
configuration. It loads the three kinds with the full-size response
capture (`full-database-response-capture.ts`). Before: `heartbeat_runs`
6,317,700 bytes, `agents` 84,336 bytes, `issues` 4,235,810 bytes. After:
under the test bounds of 20,000, 5,000 and 60,000 bytes. The decisions
are the same before and after: 50 failed runs, 40 reviews and 10
questions.
- `scopes run lifecycle attention hints to the run's task graph and
agent status to agent alerts` (I saw it fail first):
`writeExecutionControl` and `notifyExecutionRunWork` with a run id send
the hint for the task and its parent. `finalizeAgentStatus` sends the
agent-alert hint. Before, all three sent `{ companyId }`.
- `attention-retention-work` (57 tests),
`attention-retention-candidates`, `attention-service`,
`execution-control-work` and `heartbeat-agent-status-recovery` pass,
each file run alone (in `server/`, against a dedicated local `_test`
database).
- `pnpm check:module-boundaries` passes.

Not done in this pull request, and why:

- I did not reproduce `syncItems` re-syncing unchanged rows from the
retention `check`. On my fixture (300 stalled blocked tasks, 4,500 old
recovery-action and blocker retention rows, run churn), `syncItems` ran
0 times in a 180-second window and `decision_retention` reads were 0 per
second. `check` calls `syncItems` only for entries that are already due
(older than the 90-day archive delay), and a due entry is archived and
leaves the candidates. So I did not batch `syncItems` or skip checks of
unchanged rows.
- `readIssues` still selects the `issues` columns that blocker attention
needs. I did not narrow `issueService.listBlockerAttentionBatch`.

## Risks

- A producer that changes more than the run's task graph would now miss
the wider effect. Company-level changes (agent configuration, company
status, budgets) keep their company hints. The 90-day archive delay
makes a missed reload cheap: the item is reloaded on the next hint that
covers it.
- The saved task id now comes from SQL. The SQL mirrors `readRunIssueId`
for strings, absent keys and JSON null. A non-string `issueId` value now
yields no task, as before.
- No schema change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 02:17:21 -06:00
DottaandClaude Opus 5.5 bbed32450c perf(load-model): model recorded native run rates, the recorded event mix and coalesced delta batches (#21549)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - `scripts/benchmark-agent-scaling-load.ts` (#21418) estimates how
many concurrent agents one server process can carry
> - It assumed 20 text deltas/s and 1 other event/s per agent, and every
other event was an `item.completed`
> - Recorded hosted native runs show about 0.2 deltas/s and 0.5 other
events/s, with a mix in which 39% of the other events are server trace
spans that take a cheaper path
> - This pull request makes the recorded rates and mix the model's
default, keeps the old rates as an explicit stress profile, adds a burst
profile, and mirrors the delta batching of #21520
> - The benefit is a model that measures what production does, so the
next optimizations target the real per-agent cost

## Linked Issues or Issue Description

Refs #21418 (load model and plan). Refs #21520 (delta batching that the
model now mirrors).

**What existing behavior does this improve?**
The load model's default rates, its event mix and its delta batching
rule, and the plan document's measurements.

**Subsystem affected**
`scripts/benchmark-agent-scaling-load.ts`, its test
`server/src/__tests__/agent-scaling-load-model.test.ts`, and
`doc/plans/2026-11-13-thousand-concurrent-agents.md`.

**Current behavior**
The model defaults to 20 deltas/s and 1 other event/s per agent. Every
other event is a native `item.completed` with a chat-progress wake and a
checkpoint save. It cuts a delta batch on the timer even while the
previous batch commits.

**Proposed behavior**
- `PROFILE=recorded` (default): 0.2 deltas/s and 0.5 other events/s per
agent. `PROFILE=burst`: 2.5 and 3, a run's busiest minute.
`PROFILE=stress`: the earlier 20 and 1. API requests stay at 0.3/s. Each
rate can still be set on its own.
- `EVENT_MIX=recorded` (default) sends other events in the recorded
proportions. A `run.performance.span` is an ordinary append with no wake
and no checkpoint, as the server's run trace writes it.
`tool.execution.*` and `item.completed` are native appends with a
chat-progress wake, then a checkpoint save. `EVENT_MIX=<type>` sends one
type only.
- Delta batching follows #21520: deltas that arrive while a batch
commits all join one next batch.
- The report adds `profile` and `eventsIssuedByType`.

**Reason and benefit**
At the recorded rates the per-agent cost is the non-delta runner events
and their checkpoint saves, not deltas. The plan and the next
optimizations need that picture.

**Breaking changes**
None for the product. Anyone who runs the model with no options now gets
the recorded profile; `PROFILE=stress` gives the earlier load.

## What Changed

- Profiles, the recorded event mix (a 100-slot cycle per agent, offset
by agent so the fleet sends the mix at every moment), the span append
path, and the coalesced delta batches.
- The model test now checks the profile in the report and that both
server trace spans and runner events with a progress wake are in the
mix, for the recorded profile and for the stress profile with one
transaction per delta.
- The plan has a new section "Load model at recorded rates": the
recorded source (40 recent native runs on one hosted workspace, as
aggregates), the profiles, the mix and its server paths, fresh runs at
500, 1,000 and 2,000 agents plus burst and stress, and what they show.
It records the 3-connection checkpoint pool as a computed per-process
limit on a hosted database, and that SQL building in Drizzle costs about
as much main-thread time as result parsing on the hosted instance. A new
section "Considered and not done" records why `run.performance.span`
rows stay (about 3% of the statements per agent, with readers) and why
checkpoint saves per critical event only are not done now.

## Verification

- `server/src/__tests__/agent-scaling-load-model.test.ts` passes (2
tests). The new assertions fail against the previous script: it reports
no `profile` and no `eventsIssuedByType`.
- `server/src/__tests__/agent-scaling-load-model.test.ts` also checks
each profile's rates with no overrides, and that a rate set on its own
wins.
- Load model runs, recorded in the plan (60 s, main pool of 10, embedded
PostgreSQL on the same machine, each run started below a 1-minute load
average of 15): recorded rates at 1,000 agents (event-loop utilization
0.61) and 2,000 (0.85) finish every operation with no failure; 500
agents all in their busiest minute run at 0.95 with p99 lateness under
0.4 s. About 4.4 statements per agent per second at recorded rates, 35%
of them on the checkpoint pool.
- Earlier runs at load averages of 20–55 had shown one process falling
behind at 500 burst agents and at 2,000 recorded; the quieter runs after
the model fixes (event order, spans off the runner stream) do not, and
the plan says so. The checkpoint pool as a per-process limit is now
stated as a computed limit for a hosted round trip (about 400 agents in
a busy minute at 1 ms), not as measured.
- Greptile findings fixed: a waiting delta batch no longer takes deltas
sent after the next runner event, takes every delta queued when it
starts, and spans no longer join the runner event stream.

## Risks

- Low: a benchmark script, its test and a plan document. No product code
changes.
- The machine that measured the plan's table was shared; the plan marks
the rows that the machine distorted.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 23:13:07 -06:00
DottaandClaude Opus 5.5 a4c570aede feat(mcp): ask an app's form questions in the task chat and send the answer back (MCP elicitation) (#21466)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agents use connected MCP apps through the tool gateway.
> - An MCP app can ask the person a question in the middle of a tool
call (MCP elicitation). There are two kinds: a form, or a link to open.
> - Today a form question is a dead end:
> - A 2026-07-28 server's form `input_required` stops with "needs
information Paperclip can't collect yet".
> - An older server's `elicitation/create` makes a card, but the answer
is never sent back, and the server waits on an open stream.
> - Paperclip does not declare form support, so compliant servers never
ask.
> - This pull request shows the question as a card in the task chat. The
card names the app, and only a person can answer it. When the person
answers, the agent wakes. The agent's repeated call sends the answer to
the app exactly once, for both protocol eras.
> - The benefit: apps that need a choice or a detail from the person now
finish their work. Restarts lose nothing, and no call is left waiting.

## Linked Issues or Issue Description

Refs #21140 (stateless MCP client; this PR closes its "form elicitation"
follow-up). Refs #21263 (MCP Tasks; this PR shares its card code). Refs
#21389 (MCP Tasks follow-ups: the card answer validation item).

**Is your feature request related to a problem?**
When a connected MCP app asks the person a form question during a tool
call, the call stops and cannot continue. A 2026-07-28 server gets a
"needs input" error. An older server gets a card whose answer never
reaches it.

**Describe the solution you'd like**
Show the question in the task chat. Let a person answer, skip, or leave
it. Wake the agent when the question is resolved. Send the answer on the
agent's repeated call, once, with the server's exact state.

**Describe alternatives you've considered**
Holding the tool call open until the person answers. We rejected this:
it holds a stream and a run slot, and it does not survive deploys or
restarts. The plan (`doc/plans/2026-11-13-mcp-elicitation.md`) records
nine design decisions with their alternatives.

## What Changed

- **Plan:** `doc/plans/2026-11-13-mcp-elicitation.md` covers the spec,
the inventory, the decisions, the journey, tests, and risks.
- **New module `server/src/services/mcp-elicitation.ts`.** It is the
single mapping for direct calls and MCP Tasks:
- It turns each spec form field into a real card field: text with
defaults, number, yes/no, single choice (`enum` and `oneOf`), and
multiple choice (`items.enum` and `items.anyOf`).
  - It turns card answers back into typed `content`.
- Answered sends `accept`. Skip sends `decline`. Withdrawn, closed, or
expired sends `cancel`.
- It refuses forms that ask for passwords, keys, tokens, card numbers,
or one-time codes.
  - It keeps app text plain, so links in it are not clickable.
- It seals `requestState` with AES-256-GCM, bound to the company and
connection.
- **New table `tool_mcp_elicitations`** (migration 0479). It is
company-scoped, with one row per question round. It holds a keyed call
identity, the sealed state, the questions and their cards, a claim time,
and a 24-hour expiry.
- **Gateway, stateless servers:**
  - Paperclip declares `elicitation: {form, url}` on task tool calls.
- A form-only `input_required` saves the round, makes the cards, and
stops with a wait message.
- A repeated call before the person answers stops again without
contacting the app.
- After the answer, the repeated call claims the round and sends
`inputResponses` and the exact `requestState`, once.
- A failed send releases the claim. A crash mid-send lets the claim
lapse.
- **Gateway, older servers:**
  - Paperclip declares form support at `initialize`.
  - The first question gets `cancel` at once, so the server never waits.
- When the same question (same words and schema) arrives on the repeated
call, Paperclip answers it from the card.
- **Gateway, mixed rounds and retries:**
  - A form beside a URL step shows the link first.
- A repeated call that stopped for a question runs again in the same
run. It is no longer replayed by the per-run idempotency key.
- **Cards:** the title is "*App* needs your input". The resolver policy
is `human_only`. "Send to *App*" sends the answer. A user comment does
not supersede the card.
- **Activity log:** new events `tool_gateway.elicitation_sent`,
`tool_gateway.elicitation_refused`, and
`tool_gateway.elicitation_expired`. They record the action only, never
the values.
- **MCP Tasks:** task cards use the same mapping and gain the app name,
real fields, and `human_only`. A secret or an unsupported form stays
unsupported.
- **Storybook:** `Tasks and conversations/Questions and approvals/MCP
App Questions` has five states: Asking, Answered, Skipped, Expired, and
TwoQuestions. The fixtures are generated from the server mapper's
output.
- **Docs:** `doc/connections/MCP-ELICITATION.md` (linked from the
README) is new. `MCP-PROTOCOL-VERSIONS.md` and `MCP-TASKS.md` are
updated.

## Verification

- `server/src/__tests__/mcp-elicitation.test.ts` has 16 tests. They pass
on the branch.
- **Red/green:** 9 of the 9 gateway tests fail with master's
`tool-gateway.ts` and `mcp-http.ts`. The 7 pure unit tests cover the new
module.
- The tests use a fake 2026-07-28 server and a fake 2025-06-18 server
that asks over the open stream and waits for the reply.
- Covered: answer, skip (`decline`), withdraw (`cancel`), two questions
at once, a restart while waiting, a crash mid-send, another company with
the same app and arguments, expiry, a refused password field, an agent
that tries to answer (rejected), and capabilities.
- Existing suites: `mcp-tasks`, `mcp-protocol-versions`,
`tool-gateway-service`, `tool-gateway`, and `remote-mcp-connectors`. See
the results comment below.
- Storybook: the five stories render in headless Chromium, and the play
tests pass. The only console errors are the 404s from the backend-less
API calls that every story makes.
- `tsc --noEmit` is clean for `server` and `ui`. `node
scripts/storybook-catalog.mjs` passes.

## Risks

- **Older servers may start asking questions.** Declaring form support
at `initialize` can make older servers ask where they did not before.
Mitigation: the first question always gets `cancel` at once, so no call
hangs, and the agent gets a clear next step.
- **Per-call cost.** Each 2026-07-28 HTTP tool call in a task makes one
indexed lookup for a waiting round. Nothing is added before the first
token or to agent boot.
- **Migration.** One additive table with no backfill, so it is safe to
apply online.
- **Changed Tasks behavior.** A task card that is withdrawn or closed
now sends `cancel` instead of `decline`, as the spec defines. A skipped
card still sends `decline`.
- **Stale cards.** Expiry is checked only when the agent calls again. A
card can sit past 24 hours until the next call. Then it is withdrawn and
the app asks again.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code with tool use
(shell, file edits, headless browser), with extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 19:46:56 -06:00
DottaandClaude Fable 5.2 05fef9f27b perf(server): add a load model of per-agent server cost and the plan for 1,000 concurrent agents (#21418)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - One server process controls every agent run of a workspace
> - A hosted workspace slows down with about 50 agents running at the
same time
> - The target is 1,000 agents at the same time, served by several
server processes
> - This pull request adds the plan and a local load model that measures
what each running agent costs the server
> - The benefit is a measured ranking of the causes, and a repeatable
before/after number for each fix

## Linked Issues or Issue Description

**What existing behavior does this improve?**

The server controls agent runs. With about 50 runs active, requests wait
and run events fall behind.

**Current behavior**

Each text delta from an agent is one database transaction. In the load
model, 50 agents at 20 deltas per second issue about 3,900 statements
per second, and the process completes about half of the offered deltas.
Agent authentication slows from 7 ms to 464 ms because it shares the
connection pool.

**Proposed behavior**

The plan gives three phases: measure, decrease the cost per agent, then
let several server processes own runs. Each fix is one pull request with
a test and a before/after number from the load model.

**Reason and benefit**

The ranking was an estimate from code. The load model replaces the
estimate with a measurement and gives each fix a number to improve.

## What Changed

- Added `doc/plans/2026-11-13-thousand-concurrent-agents.md`: finish
line, evidence, load model result, phases, and owner decisions. It now
also has a "Load model after Phase 1" section. That section has a table
at 50, 100, 200 and 400 agents and a row at 200 agents with deltas and
leases as before Phase 1. The 400-agent delta count was measured before
the count fix and is marked as not reliable.
- Added `scripts/benchmark-agent-scaling-load.ts`: N simulated agents
call the real event append, controller checkpoint save, lease renewal,
and agent authentication code against an embedded PostgreSQL. It reports
statements for each database pool, event-loop utilization, and lateness
for each source.
- Deltas: by default the model commits deltas as the server port does
after #21460. The first text after another event commits at once. Later
deltas commit in one transaction (`appendHeartbeatRunEventBatch`) per
250 ms or 100 deltas, and the stream does not wait for the receipts.
`DELTA_BATCH_MS=0` keeps one transaction per delta, as before batching.
- Leases: by default the model renews every lease through the shared
schedule from #21444 (`startNativeSessionExecutionLeaseRenewal`,
`renewNativeExecutionLeasesNow`). `LEASE_MODE=single` keeps one
transaction per run, as before.
- The commit counter is read before the checkpoint drain, so commits
that come after the window do not count in it.
- Per-source counts freeze when the window closes. An operation that
settles later, while queued work drains, is not done.
- Batch lease mode needs `LEASE_S` of at most 20. The shared schedule
renews a remote lease on its own 20 s after its last renewal, and the
model does not count that work, so the script rejects a longer interval.
- Added `server/src/__tests__/agent-scaling-load-model.test.ts`. It runs
the model end to end in both modes: the default (batched deltas and
shared lease renewal) and `DELTA_BATCH_MS=0 LEASE_MODE=single`. It
checks that every source finishes without failures and that each pool is
counted.

## Verification

- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/agent-scaling-load-model.test.ts` on a tree that contains
#21460 and #21444: 2 of 2 pass (both modes). The test also checks that
no source reports more done than offered.
- `pnpm --filter @paperclipai/server exec tsx
../scripts/benchmark-agent-scaling-load.ts` with `AGENTS=50 SECONDS=20`
and higher agent counts. The results are in the plan.
- `pnpm check:module-boundaries` passes.
- No product code changed.

## Risks

Low risk. The pull request adds a document and a script that no product
code imports. The load model runs on loopback PostgreSQL, so its
absolute numbers are not a prediction for a hosted database. The plan
states this limit.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Claude Code, with tool use and
code execution. Subagents on the same model read the code paths.
- The two-mode load model test and the merge with master: Claude Opus
5.5 (`claude-opus-5-5`), Claude Code, with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [ ] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 16:06:40 -06:00
DottaandClaude Opus 5.5 cfa959add7 refactor(slack): correct stale status comment and add production acceptance checklist (#20840)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Slack chat connector lets people talk to their agents in Slack
> - The connector is close to production, but its live evidence is out
of date
> - The September live results use old revisions, and the October setup
run did not test follow-up, reauthorization, or removal
> - Cloud managed setup has never run against real Slack, and its docs
say not to enable it until it has
> - This pull request adds one checklist for a fresh live pass on
current master, and corrects one stale code comment
> - The benefit is a clear finish line for Slack production readiness on
self-hosted and Cloud

## Linked Issues or Issue Description

**Issue type**
Missing documentation.

**Where is the issue?**
`doc/connections/SLACK-AUTOMATIC-SETUP.md`,
`doc/plans/chat-adapters/2026-09-05-slack-live-qualification-result.md`,
`doc/testing/slack-cloud-qualification.md`, and
`server/src/services/chat-channels.ts`.

**What's wrong?**
The Slack live qualification is spread over three documents. Their
evidence uses older revisions. No document lists what a production pass
must cover for Cloud managed setup. A code comment says the manifest
does not request `assistant:write`, but the manifest does request it.

**Suggested fix**
Add one dated acceptance checklist with pass conditions per row. Correct
the comment.

## What Changed

- Added `doc/plans/2026-11-11-slack-production-acceptance.md`. It has
sections for self-hosted automatic setup, manual and existing apps,
conversation behavior, Cloud managed setup on staging, and distribution
questions (Marketplace-only workspaces, non-Marketplace rate limits,
scope justification, zero-copy).
- Corrected the comment in `server/src/services/chat-channels.ts`. It
now says that Slack shows the working state through
`agents.sessions.setStatus` and the lifecycle reply. No code behavior
changed.

## Verification

- Documentation and a comment only. No runtime behavior changed.
- I checked the doc links and the referenced files against current
master.

## Risks

- Low risk. The change is documentation and one comment.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
web search.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [ ] I have run tests locally and they pass
- [ ] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 12:54:50 -06:00
DottaandClaude Opus 5.5 944b1c64d7 feat(mcp): keep long provider calls durable with the MCP Tasks extension (#21263)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents use connected apps through Paperclip's MCP client in the tool
gateway
> - Some provider actions run for minutes: CI runs, deploys, batch jobs
> - Today such a call is lost if it outlives the tool timeout or if
Paperclip restarts or deploys mid-call, and a retry can start the work a
second time
> - Recovery must just work, and the target rate of runs that need a
manual retry is 0%
> - This pull request implements the MCP Tasks extension
(`io.modelcontextprotocol/tasks`, 2026-07-28) in the MCP client
> - The benefit is that a long provider call becomes a durable task: it
survives timeouts, restarts, and deploys, and it is never started twice

## Linked Issues or Issue Description

**Problem**
A long provider tool call holds an HTTP request open. When it passes the
tool timeout, or when Paperclip restarts or deploys during the call, the
result is lost. The agent then often repeats the call, and the provider
does the work again.

**Proposed solution**
Implement the client side of the MCP Tasks extension. Declare it on
stateless tool calls. Store each returned task durably, wait on it
during the call, keep polling it in the background after the call ends,
and let the same call attach to the stored task instead of sending
`tools/call` again.

**Why it matters**
CI, deploy, and batch actions are where agents lose the most work today.
A durable task handle makes these actions safe across timeouts,
restarts, and deploys, without a person.

Refs #21119, Refs #21140, Refs #21219 (the MCP Apps, stateless MCP
client, and MCP Skills work this builds on).

## What Changed

- **Off by default:** MCP Tasks are behind the new instance experimental
setting `enableMcpTasks` ("Durable app actions (MCP Tasks)"), default
off. When it is off, Paperclip never declares the extension, nothing
attaches to or polls tasks, pending rows do not block idle sleep, and
behavior matches master. A test covers the off state.

- `mcp-http.ts`: the tasks extension is declared per request, only on
HTTP `tools/call` and `tasks/*`. Discovery, listings, and stdio servers
do not declare it. `tasks/*` requests put the task ID in `Mcp-Name`.
- New `mcp-tasks.ts`: parses task results, bounds the server's timing
hints, sends `tasks/get`, `tasks/update`, and `tasks/cancel`, and seals
task IDs at rest (AES-256-GCM, bound to company and connection; a keyed
hash gives uniqueness).
- New company-scoped table `tool_mcp_tasks` (migration `0475`,
online-compatible). It links the task to the call, run, task,
connection, and grant. It stores no provider result.
- `tool-gateway.ts`:
- A returned task is recorded, then polled up to the normal tool
timeout. Its result goes through the same validation, redaction, and
recording as a direct result.
- Past the timeout, or when the caller disconnects, the agent gets
`provider_task_pending` ("call again with the same arguments"). The task
stays durable.
- A call with the same endpoint, tool, full arguments, grant, agent,
task, and actor attaches to the undelivered task. A retry with the same
idempotency key also attaches, and it can never start different work
under that key.
- Input: URL requests show the provider link (Paperclip never answers
them). Form requests become a question card; only the person's answer is
sent with `tasks/update`. Other requests stop with a clear "needs input"
error.
- Stopping the run sends `tasks/cancel` and marks the task cancelled. A
caller that only disconnects keeps the task.
- `failed` becomes a tool error with the provider's message. An unknown
task or a passed TTL becomes `provider_task_expired`. Every task stops
being tracked at its TTL, or after 7 days at most.
- Audit entries: `tool_gateway.task_started`, `task_input_sent`,
`task_cancelled`.
- New `mcp-task-work.ts`, registered at startup with the shared deadline
scheduler: it resumes polling pending tasks after a restart, cancels
tasks whose task closed or run was stopped, and wakes the assignee once
when a task settles after its run ended.
- Docs: `doc/connections/MCP-TASKS.md`, linked from
`doc/connections/README.md`.

Background work (per `rules/sleep-aware-background-work.mdc`): trigger
is a task row write or release (`mcp-tasks` queue) and one startup pass.
The deadline is the earliest `next_poll_at`. Cadence is the server's
`pollIntervalMs` within 5 s – 5 min, with backoff; 5 min while waiting
for a person. It is quiet when no row has a next poll. This is the
narrow external-source exception: providers give no wake Paperclip can
receive, and polling stops with the task.

## Verification

- `server/src/__tests__/mcp-tasks.test.ts`: 20 tests with a fake
2026-07-28 server that returns tasks only to a client that declares the
extension and checks `Mcp-Name`. They cover:
- polling to completion, with the result identical to a synchronous
call;
  - timeout, then an attaching call with no second `tools/call`;
  - a keyed retry, and a reused key with different arguments (refused);
- restart mid-task: a fresh gateway with no in-memory state polls the
row, wakes the agent once, and the recovered run gets the result once
with no second `tools/call`;
  - no wake while the run is still going;
- URL input (link, never answered), and form input (question card, then
`tasks/update` with the answer);
- cancellation by the run, by closing the task, and no cancellation when
only the caller leaves;
- failure, an unknown task, and TTL expiry (ends in under 3 s, no hang);
  - another agent never attaches;
- synchronous calls unchanged and older servers never see the extension.
- Red/green: with attach disabled, the timeout, keyed-retry, and restart
tests fail (3). With `tasks/cancel` disabled, both cancellation tests
fail (2). Before `tools/call` declared the extension, 13 end-to-end
tests failed. All pass with the change.
- Existing suites pass: `mcp-protocol-versions`, `mcp-http`, `mcp-apps`,
`mcp-skills`, `server-startup-feedback-export`, `tool-gateway`,
`tool-gateway-discovery-http`, `remote-mcp-connectors` (342 tests).
- `tsc --noEmit -p server` is clean. `pnpm --filter @paperclipai/db
check:migrations` passes (the migration is classified
online-compatible). Snapshots pruned to five.
- Agent boot and time-to-first-token: no change in the boot or
first-turn path. The only new per-call cost is one indexed lookup on
stateless tool calls.
- Not run: a live provider that returns tasks. No public MCP server that
returns tasks was available without new credentials.

## Risks

- Shipped off by default. Review findings that only affect the
setting-on path after this merge are tracked as follow-ups.

- Migration: a new table with indexes and foreign keys to `companies`,
`tool_invocations`, and `tool_connections`. Online-compatible.
- Behaviour change only for servers that return tasks, which they may do
only to clients that declare the extension. Older servers and stdio
servers are unchanged.
- A task that settles while nobody is waiting stays attachable for 1
hour, so a deliberate repeat of the same call within that hour collects
the earlier result instead of starting new work.
- The wake after a background task settles uses the generic task wake;
the agent must call the tool again to collect the result.
- Follow-ups: `notifications/tasks` via `subscriptions/listen`; sampling
and roots input requests; showing a long action's progress in the task
chat.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, file edits, local test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 08:32:06 -06:00
DottaandClaude 3914d0c94a test(chaos): add DF1, a sandbox disk-full fault on the final workspace save (#20810)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart chaos harness (`tests/restart-chaos/`) runs real agent
tasks and breaks the control plane on purpose, so failures show up
before staging users see them
> - A census of 48 hours of staging failures found runs whose result
save-back failed because the sandbox disk was full (class F). The
harness never filled a sandbox disk, so it missed that class
> - #20758 made a full-disk save end recovery at once with cause
`sandbox_disk_full` and a Retry. Nothing in chaos exercised that path
end to end
> - A real full disk cannot be made on demand. The owner chose a
test-only hook that makes the save report a full disk, not a small disk
> - This pull request adds the hook at the save's sandbox boundary and a
DF1 fault that arms it, clears it, and grades the result
> - The benefit is that every wave can now check the disk-full rest, the
Retry offer, and that Retry works after space is freed

## Linked Issues or Issue Description

Refs #20758 (product behaviour under test), #20678 (finalization hands a
full-disk copy-back to recovery)

**What is the problem?**
The chaos harness had no fault for a full sandbox disk during result
save-back. Staging saw this failure (census class F in
`doc/plans/2026-11-11-staging-failures-vs-chaos-coverage.md`), and chaos
could not reproduce it.

**What is the expected behaviour?**
A wave can schedule DF1. While DF1 is armed, the task's final workspace
saves fail as a full disk. When DF1 is cleared, saves succeed again. The
task must rest with `sandbox_disk_full` and a Retry, and that Retry must
finish the task.

**Why does it matter?**
Without a fault, a regression in the disk-full path (no Retry, a wrong
cause, or a Retry that does not work) reaches staging users before chaos
sees it.

## What Changed

- `server/src/services/sandbox-disk-full-fault.ts` (new, test-only):
`sandboxDiskFullFaultTarget` wraps the target of a final save only when
the harness armed that save. The capture commands then exit 1 with the
exact message the in-sandbox script writes on ENOSPC, and the provider
download fails with `No space left on device`. The gate follows
`@paperclipai/shared/crash-points`: `PAPERCLIP_DISK_FULL_FAULT_CONTROL`
is read once at import and ignored under `NODE_ENV=production`.
Disarmed, the hook is one comparison and returns the save's own target.
- `server/src/services/remote-workspaces.ts`: the capture call in
`saveRemoteWorkspace` passes its target through the hook. No other
product change. `workspace-recovery.ts` is not touched.
- `tests/restart-chaos/disk-full-faults.ts` (new): the DF1 kind, the
control-file seam (atomic writes), the rest evidence
(`activeRecoveryAction.evidence.systemRecovery.failure.cause.label`),
the `disk_full_fault` T check, and the DF1 R3 rule (one expected Retry
that must finish the task).
- Harness wiring: `--faults DF1` (refused without `--env daytona`, with
a storm, or with PV faults), no scheduled restart, arm at launch, clear
at the first rest just before the Retry click, receipts in
`<wave>/disk-full-faults.jsonl`, and ledger fields `diskFull: {
faultedSaves, restedDiskFull, retryOffered, retryWorked }`.
- Docs: a "Sandbox disk full (DF1)" section in
`tests/restart-chaos/README.md`, and the census plan marks DF1 as built.

## Verification

- `cd server && pnpm exec vitest run
src/services/sandbox-disk-full-fault.test.ts`: the gate is off by
default and under `NODE_ENV=production`, the save's own target comes
back when disarmed, only a listed final save is faulted, the product
capture turns it into the named disk-full failure, and clearing the file
disarms the next save.
- `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts
tests/restart-chaos/disk-full-faults.test.ts
tests/restart-chaos/grader.test.ts tests/restart-chaos/scheduler.test.ts
tests/restart-chaos/provider-faults.test.ts`: planning, the control
file, and grading.
- `node scripts/typecheck-test-projects.mjs tests/restart-chaos` and the
server `tsc --noEmit`: clean.
- Live Daytona waves (Codex, `--storage s3`, image content `96c4339f…`,
run through the two-slot queue):
- `w-20261111T131314-s9411` (B1): the hook was armed in the server but
no save reached it. B1 uses a local worktree synced into the sandbox,
not a remote workspace. Graded `not_applied` as designed. The harness
now requires `--storage s3` and the docs point at R1.
- `w-20261111T131941-s9412` (R1): 3 faulted saves. t01 rested `blocked`
with `sandbox_disk_full`, `user_action_required`, board owner, API
`canRetry: true`. t02 was cleared too early, while recovery still
checked its source; that recovery then saved and the task finished.
Fixed in 626f552e (wait for the disk-full verdict).
- `w-20261111T134316-s9413` (R1, after the fix): 2/2 tasks rested with
`sandbox_disk_full` and API `canRetry: true`. In both, the board showed
**"Recover saved files"** and no Retry button, so the UI Retry click
failed (`ui_retry_button_missing`). Retry-worked is not shown. The wave
was interrupted by SIGTERM during the post-rest file reads, before final
grading.
- Local wave: not run. A local run has no sandbox workspace save, so DF1
cannot fire there; the harness refuses `--env local`.

## Risks

- Low risk for production. The hook reads its variable once at startup
and is ignored under `NODE_ENV=production`. Without the variable it does
one comparison and returns the same target object.
- The hook fails every sandbox command of an armed capture, including
the read-only fallback's inventory. A real full disk can let that
inventory succeed and fail later at the download. The outcome is the
same (`sandbox_disk_full`), but the fallback's inventory step is not
exercised under DF1.
- DF1 runs only with `--env daytona --storage s3` and a remote workspace
(R1). A local run has no sandbox workspace save, so the harness refuses
DF1 there. DF1 is refused in `--matrix` waves, and T1's restart check is
skipped for DF1.

## Model Used

- Claude Opus 5.5 (Anthropic, model ID `claude-opus-5-5`), in Claude
Code with tool use (shell, file edits, GitHub CLI).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-11 09:57:59 -06:00
DottaandClaude 62b7733161 test(restart-chaos): long mid-tool K15 outages, Runner evidence, and TM1 stalls on Daytona (#20711)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The restart-chaos harness (`tests/restart-chaos/`) restarts or
stalls the control plane while real agent tasks run.
> - The staging census (#20679,
`doc/plans/2026-11-11-staging-failures-vs-chaos-coverage.md`) found two
open classes that the harness could not create. In class R, a deploy
lands mid-turn, the server stays down for 4–7 minutes, and recovery then
finds the remote Runner gone
(`runner_remote_process_identity_unavailable`). In class L, the server
stalls for about 100 s without a restart, and legacy runs that are still
`preparing` end up waiting for a manual Retry.
> - K15 restarted at once. No Daytona workload kept a tool running
across the fault. TM1 was local-only with a fixed range. Nothing
recorded what the remote Runner did while the controller was away.
> - This pull request adds a K15 outage hold, a mid-tool workload,
sandbox-side Runner evidence, and TM1 stalls on Daytona with a
configurable hold.
> - The benefit: these staging failure shapes can now be reproduced, and
each wave records the Runner's own state.

## Linked Issues or Issue Description

Refs #20679

**What happened?**
On staging, two failure shapes had no matching chaos scenario:
- Class R: a deploy landed mid-turn and the server stayed down for 4–7
minutes. Recovery then found the remote Runner gone. Nothing captured
why the Runner exited.
- Class L: a 100 s server stall without a restart lapsed every
controller lease. Legacy `preparing` runs then waited for a manual
Retry.

**What did you expect?**
The harness should create a long mid-turn deploy outage and a stall
without a restart on Daytona. It should also save the sandbox-side
Runner evidence.

**Steps to reproduce**
1. Use the census report and the staging run records listed in it.
2. Compare them with the harness options before this change: K15 has no
hold, TM1 is local-only, and there is no Daytona tool workload.

## What Changed

- `--k15-hold-seconds <n|min-max>` (0–600). A K15 stops the server with
SIGTERM, keeps it down for the seeded hold, then starts the next deploy
generation. The range is drawn only for K15 cells, so existing seeds
keep their plans. Storms use it too.
- New opt-in workload `T1`. The agent runs one shell loop in the
foreground for about 100 s, then saves `TICKS-<nonce>=50`. Its fault
window, 35–80 s after launch, lands mid-tool.
- New `daytona-runner-logs.ts`, a read-only probe of each wave sandbox
through the Daytona toolbox. It runs only on sandboxes with the wave's
labels, and only on `started` ones. It records every process, memory,
disk, `dmesg` OOM lines, and the tails of `runnerd.stderr.log`,
`runner-process.identity` and `runner-state.json`. It redacts known
secrets. It runs right after a K15 stop, just before the next deploy
starts, and at wave end. Output goes to `results/<wave>/daytona-runner/`
and to `runner_logs_captured` in the timeline. The probe never stops,
starts or deletes a sandbox, and a failed capture is recorded instead of
thrown.
- TM1 now runs on `--env daytona`, and `--tm1-hold-seconds <n|min-max>`
(10–600) replaces its default 95–150 s range with the same single draw.
TM2 and TM3 stay local.
- Tests: `k15-long-hold.test.ts` and `tm1-stall.test.ts`. The README is
updated.

## Verification

- `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`:
45 files and 531 tests pass.
- `node scripts/typecheck-test-projects.mjs tests/restart-chaos`:
passes.
- Daytona waves, all with T1 and K15:

  | Wave | Revision and image | Hold | Result |
  | --- | --- | --- | --- |
| `w-20261111T015441-s9310` | master 8f6ca25976, image `bc7ca92d…` |
267–408 s, OpenCode | 4 of 4 pass |
| `w-20261111T022637-s9311` | master 8f6ca25976, image `bc7ca92d…` | 300
s, OpenCode and Claude | 4 of 4 pass. Every stop capture shows the T1
loop running. |
| `w-20261111T025307-s9320` | master a1ec05a1a4, image `09321542…` |
460–482 s, past the Runner's 7-minute reconnect grace | 4 of 4 pass |

- In every pre-restart capture, the same `runnerd` PID was alive for the
whole outage (etimes 491–549 s in the last wave). Every sandbox was
`started`. There was no `transport_reconnect_grace_exceeded` and no OOM.
- So the class R Runner loss does not reproduce on current master with
the e2e image. The census document lists the remaining differences from
staging.
- TM1 Daytona waves on master a1ec05a1a4 found a **new product gap**
(RED; the harness only reports it):
- `w-20261111T034105-s9330` (TM1 at `--fault-after-seconds 10`): 2 of 3
pass. The failing OpenCode run was stalled at
`native.process_start_requested`. Afterwards it waited forever with
`controller_still_alive` / `restartKind: hard`, where the controller it
waited for was its own live server.
- `w-20261111T040439-s9331` (TM1 mid-tool, T1): 0 of 3 pass. After the
stall the run had no further events and stayed `running` for more than
20 minutes, with no Retry.
- Review fixes in 3540bfe700: matrix cells take both holds, T1's
`fault_crossed_tool` check, and one 45 s deadline per Runner capture.
The suite is now 45 files and 531 tests.

## Risks

- Low risk. This change is test harness and docs only, with no product
code.
- The Runner probe executes one fixed read-only command in this wave's
own sandboxes through the Daytona toolbox. It never targets fleet
sandboxes, because it filters on the wave's company and environment
labels.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, file edit, GitHub CLI).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-11 00:29:06 -06:00
DottaandClaude 3fdcc2a28c fix(runner): resend an ACPX turn whose start was cut off before the prompt left (#20675)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native ACPX run (for example Claude) sends `turn.start` to its
Runner. The Runner asks the ACPX sidecar to start the turn, and the
sidecar sends the prompt to the provider
> - A start can fail without a definite answer. A host freeze can
outlast the sidecar request deadline, or the sidecar or Runner can die
during the start
> - The Runner then saved the session `closed`. Every same-run resume
failed with `ACPX session is not open`, and the task needed Retry
> - The Runner cannot tell from its own state whether the provider has
the prompt. Resending could duplicate it. A continuation could lose it
> - This pull request adds a durable record that answers that question:
a sidecar prompt-dispatch journal, written before the prompt reaches the
provider
> - The benefit is that a start cut off before the prompt left resends
the prompt exactly once, with no Retry. Every case the record cannot
prove keeps today's behavior

## Linked Issues or Issue Description

**What happened?**
Local restart-chaos wave `w-20261110T185315-s9107`, task t05 (claude,
workload C1, fault TM2). A follow-up run reopened the parked Claude
session and sent `turn.start`. The sidecar first starts Claude and loads
the conversation; the prompt goes out only after that. After 24 s
without a reply, the harness froze the server, Runner, sidecar and
Claude (SIGSTOP) for 33 s. On thaw the Runner's 30 s sidecar deadline
had passed. It reaped the sidecar and saved the session `closed`. Three
same-run resumes failed with `PRP command turn.start failed: … ACPX
session is not open`, recovery stopped with
`native_session_retry_exhausted`, and a person had to click Retry.

**Expected behavior**
When a durable record proves the prompt never reached the provider, the
run resends it once by itself. When the record cannot prove it, the run
behaves as today.

**Steps to reproduce**
1. On master, run `cargo test --manifest-path
packages/paperclip-runner/runner/Cargo.toml -p paperclip-runner-core
--test acpx_park_preparation cut_turn_start`.
2.
`a_start_cut_off_before_the_prompt_left_reopens_and_sends_the_prompt_once`
fails: the session is `closed` and the next start fails with `ACPX
session is not open`.

Refs #20507, #20645, #20674

## What Changed

- New file contract in
`protocol/provider-schemas/acpx-sidecar.schema.json`:
`turnDispatchJournalFile` (`paperclip-turn-dispatch.jsonl`) and
`turnDispatchJournalRecord` (`received` / `dispatched`). The generator
emits the file name and schema into TypeScript and Rust, so the pack
contract digest covers them.
- Sidecar (`turn-dispatch-journal.ts`, `codex-runtime-adapter.ts`,
`runtime-host.ts`, `acpx-runtime-sidecar.ts`): when the runtime port
starts a turn, before any provider work, it truncates the journal and
writes `received`. ACPX calls `onAcpMessage("outbound")` synchronously
before it writes a message to the provider. On the turn's first
`session/prompt` request the port appends `dispatched` and fsyncs it
there, so the record exists before the bytes leave. A failed dispatch
write aborts the prompt write. A failed `received` write removes the
file and runs the turn unjournaled.
- Runner (`acpx_turn_dispatch.rs`, `acpx_provider_backend.rs`): the
journal is read only after the provider lifetime fence proves that the
sidecar and provider are gone. A lone complete `received` record for the
turn proves the prompt never left.
- A start that failed in flight saves the session `prepared` with no
active turn and returns `ACPX turn start failed before its prompt
reached the provider`. The run's next `turn.start` reopens the
conversation and sends the prompt once. Until the proof settles, the
saved state stays `turn_starting`, so a Runner killed during the proof
goes through the restore check below.
- A Runner killed while the session was `turn_starting` restores the
session idle (`session.resumed`, no `turn.failed`). The same-run resume
then starts the turn once instead of sending a continuation that would
lose the prompt.
- A `dispatched` record, another turn's record, a torn or malformed
file, or no file keeps today's behavior (closed session, or lost-turn
recovery).
- Fake sidecar: writes the same journal and adds `hang-before-dispatch`
/ `hang-after-dispatch` faults.
- Docs: `packages/paperclip-runner/docs/durable-recovery.md` ("ACPX turn
starts cut off before the prompt left"). The design note is
`doc/plans/2026-11-10-durable-turn-start-and-tool-call-records.md` (in
#20674).

## Verification

- Red on master, green here: `cargo test -p paperclip-runner-core --test
acpx_park_preparation cut_turn_start` (4 tests: cut before dispatch
reopens and sends once; cut after dispatch stays closed and never
resends; Runner killed before dispatch restores idle; Runner killed
after dispatch or with no journal reports the lost turn). The two
"before" cases fail on master.
- `cargo test -p paperclip-runner-core`: all pass. `cargo fmt --check`
clean.
- `pnpm --filter @paperclipai/paperclip-runner exec vitest run
src/drivers/acpx/turn-dispatch-journal.test.ts
src/drivers/acpx/codex-runtime-adapter.test.ts`: 121 passed. One test
uses a real ACPX runtime and an ACP peer that refuses a prompt arriving
before its `dispatched` record; it fails if the hook is removed.
- `node scripts/generate-acpx-sidecar-contract.mjs --check`, `node
--test scripts/contracts/contract-fingerprints.test.mjs`, and `tsc -p
tsconfig.test.json` pass.
- Time to first token, measured with the same open/write/fsync sequence
on this Mac (APFS, load average ~11, 300 samples): `received` p50 0.24
ms, p95 0.59 ms; `dispatched` (with fsync) p50 4.7 ms, p95 11.5 ms.
Against a Claude TTFT budget of 9.5–16.5 s p50 this is under 0.1%. No
wait, round trip or file transfer is added. Recovery work (fence,
journal read) runs only after a failed start.
- Live restart-chaos wave and runner image: see the comments on this
pull request.

## Risks

- A false "not dispatched" proof would duplicate a prompt. It needs a
lone, complete `received` record for this exact turn ID, read after the
fence proves the sidecar is gone. `dispatched` is fsynced before the
prompt write, and a failed write aborts the send. A failed `received`
write removes the file, so it cannot leave a lone record.
- Upgrade safety: saved provider state gains no field (its decoder
denies unknown fields), and only lifecycles an older Runner already
handles are written. An older Runner ignores the journal file. State
saved by an older Runner has no journal, which reads as unknown and
keeps today's behavior. A new Runner with an older sidecar never sees a
`received` record, so no proof forms.
- The changed sidecar schema changes the pack contract digest, so
Daytona needs a new runner image. Older image packs are not reused.
- A failed start now waits for the provider lifetime fence (at most 5 s,
usually immediate) before it settles. This is only on a failure path.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell,
file editing) in Claude Code. Extended reasoning was used.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-10 20:15:48 -06:00
DottaandClaude 52160eba35 test(restart-chaos): K15 redeploy, F1 workspace symlink, and the staging failure census (#20679)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The restart-chaos harness (`tests/restart-chaos/`) restarts the
control plane while real agent tasks run. It grades whether every task
finishes without a human Retry.
> - The hosted staging stack deploys about every 40 minutes. In 48
hours, 276 runs failed and 63 tasks stayed blocked. About a quarter of
the failures came within 10 minutes of a deploy. The harness passed the
same areas.
> - A hosted deploy replaces the server container. The new server runs
in another process space and has a new deployment id. The PID-based "the
earlier owner exited" proofs never apply across that boundary. A harness
restart on one machine keeps the process space, so every PID proof
succeeds and the hosted recovery path is never tested.
> - This pull request adds K15, a redeploy fault. It also adds the
staging census that shows the gap.
> - The benefit: chaos waves can now exercise the recovery path that
staging runs after every deploy, on local and Daytona.

## Linked Issues or Issue Description

**What happened?**
Many staging tasks that a deploy crossed ended blocked. Examples are
`native_remote_recovery_lease_mismatch`,
`runner_remote_process_identity_unavailable`,
`prior_owner_terminal_state_indeterminate` and `PRP provider resume
state is unavailable`. The restart-chaos Daytona waves passed for the
same adapters.

**What did you expect?**
The chaos harness should reproduce the deploy conditions that staging
has, so that these failure classes show up in a wave first.

**Steps to reproduce**
1. Compare the staging run records' `controllerProcess.hostHash` and
`deployment.deploymentId` across a deploy. Both change on every deploy.
2. Compare the same fields across a harness K2 restart. Both stay the
same.
3. Read `controllerProcessExited` and `sameProcessSpace` in
`server/src/services/controller-process-identity.ts`. A different
`hostHash` means that no PID proof applies.

**Additional context**
The full census, the failure classes and the per-class harness gaps are
in `doc/plans/2026-11-11-staging-failures-vs-chaos-coverage.md`. The
task identifiers in it are neutral labels.

## What Changed

- New fault `K15` (redeploy), in `tests/restart-chaos/redeploy.ts` and
`redeploy-preload.mjs`. A K15 wave gives every server an emulated host
name through a `--import` preload, which changes the process-space hash.
It also gives Railway-shaped deployment variables with the instance home
as the exclusive volume. A K15 stops the server with SIGTERM and starts
the next generation, with a new host name and deployment id.
- `supervisor.ts`, `main.ts`, `config.ts` and `types.ts`: select K15
with `--faults K15`. It works on `--env local` and `--env daytona`. The
receipt records `redeploy` and the timeline records `redeploy_switched`.
Config refuses K15 together with MS1–MS3.
- New opt-in workload `F1` (`workloads.ts`): the agent keeps an absolute
symlink in its repository workspace. A staging save-back refused such a
link and blocked the task. The oracle needs `done` and the saved notes
file, with no Retry. `f1-symlink.test.ts` covers the prompt, the grading
and the option planning.
- `redeploy.test.ts`: identity derivation, the environment, the argument
order, a real child process that proves the preload changes both CJS and
ESM `hostname()`, option parsing and plan detection.
- `tests/restart-chaos/README.md`: a K15 section and a row in the
"Pieces" table.
- `doc/plans/2026-11-11-staging-failures-vs-chaos-coverage.md`: the
48-hour staging census, 12 failure classes, the deploy timeline, the
reason the harness missed each class, and proposed scenarios.

## Verification

- `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts
tests/restart-chaos/redeploy.test.ts
tests/restart-chaos/scheduler.test.ts
tests/restart-chaos/repeated-restarts.test.ts`: 68 tests pass. The full
restart-chaos unit suite passes: 43 files, 516 tests.
- `node scripts/typecheck-test-projects.mjs tests/restart-chaos`:
passes.
- Live Daytona waves on 8f6ca25976, image
`paperclip-daytona-runner@sha256:bc7ca92d…`:

  | Wave | Faults and cells | Result |
  | --- | --- | --- |
| `w-20261111T002313-s9301` | K15, opencode and claude, A1 and C1, reuse
lease, 6 tasks | 5 of 6 pass. t05 (OpenCode, C1) failed R5
`silent_dead_run`: 183 s without evidence after the redeploy, then
succeeded. |
  | `w-20261111T004241-s9301` | K2 control, same seed | 6 of 6 pass |
| `w-20261111T005008-s9302` | K15, warm sandbox, comment trigger, 4
tasks | 4 of 4 pass |
| `w-20261111T010000-s9303` | K15, opencode only, 8 tasks | 8 of 8 pass
|
  | `w-20261111T011044-s9303` | K2 control, same seed | 8 of 8 pass |
| `w-20261111T012523-s9304` | F1 with K2, codex and opencode, `--storage
s3` | 2 of 2 pass |

- Every K15 fault switched the generation, with a new host name and
deployment id on the same volume. The server was ready again in 14–24 s.
- K15 does not yet reproduce the staging refusal codes on current
master. The census document explains the open conditions, such as a code
change at the deploy.

## Risks

- Low risk. The change is in the test harness and docs only. No product
code changes.
- K15 is an emulation. It changes the host name only in the server
process (and forked Node children). It does not emulate a new kernel
boot or PID namespace. The cloud stop-proof service is not emulated, so
stop proofs that need it stay unsupported, as on a self-hosted Railway
deploy. The README lists these limits.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, file edit, GitHub CLI).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-10 20:05:17 -06:00
DottaandClaude e33ede9a8a fix(native): settle a restart-orphaned write_document from its transactional receipt (#20674)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run calls Paperclip tools such as `write_document`. The
runner saves each tool input durably before the server runs the tool,
and the server saves a result receipt
> - A control-plane restart can cut a `write_document` call after the
runner saved its input and before any result was saved
> - Recovery then has no proof of the outcome. A call from an earlier
server process settles as "unknown outcome" (#20628). A call the current
server process admitted stays pending, close cannot prove that the
provider drained, and the session quarantines
(`native_session_cleanup_quarantined`). A person must click Retry
> - `write_document` saves its revision and its idempotency receipt in
one database transaction, under the run and issue row locks. The
revision is also compare-and-swap on `baseRevisionId`. So the receipt is
a durable outcome record: present means applied, absent means not
applied
> - This pull request makes recovery use that proof. Recovery runs the
same receipt-guarded write. It replays the saved result, or it applies
the write exactly once
> - The benefit is that an orphaned `write_document` settles with its
real result. The document is never written twice, and the run does not
need Retry for this call

## Linked Issues or Issue Description

**What happened?**
Daytona restart-chaos wave `w-20261110T174930-s9205`, task t01 (codex,
workload C1, fault RR2). A restart cut a `write_document` call. The
close settlement showed `semanticTools.pending: [{ operationId:
"write_document", callId: "exec-8b79ab78-…" }]` and `providerDrained:
false`. The session quarantined with
`native_session_cleanup_quarantined`, and the task needed Retry. Since
#20628, a call that an earlier server process admitted settles as
`tool_call_interrupted_by_restart` with an unknown outcome. The agent
then has to guess whether the write applied.

**Expected behavior**
When a durable record proves the outcome, recovery uses it. A write that
applied returns its saved result. A write that did not apply runs once.
When the record cannot prove it, the call stays as it is today.

**Steps to reproduce**
1. On master, run `pnpm --filter @paperclipai/server exec vitest run
src/services/native-runtime/paperclip-runner-tool-authority.test.ts -t
"orphaned"`.
2. `authority.recover()` returns `null` for a `write_document` that has
no receipt, so the call stays unsettled.

Refs #20628, #20639, #20507

## What Changed

- `PaperclipRunnerToolAuthority.recover()` now settles `write_document`
through its transactional receipt (`#recoverTransactionalWrite`). It
runs the same receipt-guarded mutation as an ordinary call. If the
receipt exists, it returns the saved result and writes nothing. If no
receipt exists, it applies the write once.
- A definite refusal that rolls the transaction back (a 4xx such as a
stale `baseRevisionId`, a reused idempotency key, invalid input) settles
as a tool error. Any other failure leaves the call unsettled, as before.
- The set of operations with this proof is explicit:
`TRANSACTIONAL_RECEIPT_OPERATIONS` contains only `write_document`. Other
tools keep their current recovery rules.
- New test: an orphaned `write_document` with no receipt applies once; a
second recovery and a late original call replay the receipt; an applied
write recovers its saved result; a stale base settles as a tool error
with no change.
- Updated test: a reused idempotency key with other input now settles as
an `paperclip_runner_tool_idempotency_conflict` tool error instead of
staying pending. The receipt proves that this call can never apply under
that key.
- Docs: `packages/paperclip-runner/docs/durable-recovery.md` (semantic
operations that outlive a turn) and a new design note,
`doc/plans/2026-11-10-durable-turn-start-and-tool-call-records.md`. The
note also covers the second change (an ACPX turn start cut off before
the prompt reached the provider), which comes in a separate pull
request.

## Verification

- Red on master, green here: `pnpm --filter @paperclipai/server exec
vitest run
src/services/native-runtime/paperclip-runner-tool-authority.test.ts` (43
passed).
- `pnpm --filter @paperclipai/server exec vitest run
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`:
178 passed. One unrelated ACPX timing test failed once under load and
passed when run alone.
- Server typecheck: `npx tsc --noEmit -p server/tsconfig.json` passes.
- `pnpm verify` exits 2 (a runner doc changed), so required CI supplies
the broad coverage.
- Live restart-chaos wave: see the comment on this pull request.

## Risks

- Recovery can now apply a write that never applied before the restart.
This is the call the agent made, applied once. The write is
compare-and-swap on `baseRevisionId`, so it cannot overwrite a revision
that someone wrote after the agent read the document. That case settles
as a conflict error.
- Upgrade safety: no saved shape changes. Receipts saved by the previous
release have the same shape. A server on the previous release keeps its
current recovery for these calls.
- Time to first token: no change. This code runs only during recovery.
- No runner source changes, so no new runner image is needed.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell,
file editing) in Claude Code. Extended reasoning was used.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-10 19:05:15 -06:00
DottaandClaude 9296c74e8d fix(native): let a restart claim take back its lapsed lease at budget admission (#20650)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - After a control-plane restart, restart recovery claims an
interrupted native run (`resume_dead_runner`) and grants a 20-minute
lease. Budget admission then re-checks that the claim still owns the run
before the resumed run may start provider work.
> - A host sleep, or the restart-chaos fault TM2, can make that stored
lease look expired while its owner is alive. #20625 lets the owner take
back such a lease when the row still shows exactly its own claim. That
rule did not cover budget admission.
> - So a claim whose lease lapsed before dispatch was refused with
`Native recovery no longer owns provider dispatch`. The run failed with
`adapter_failed`, recovery held it with
`native_continuation_requires_reconciliation`, and the user had to click
Retry.
> - This pull request applies the #20625 rule in budget admission: the
claim takes its lease back through the existing re-acquire condition.
> - The benefit is that a run resumed after a restart survives a time
jump before dispatch with no Retry. Every fence and every budget check
stays the same.

## Linked Issues or Issue Description

Refs #20625, #20474

**What happened?**
Restart-chaos wave L7 (seed 9107, local) on a branch with #20625, task
codex B5 TM2. A TM3 restart claimed the run (`resume_dead_runner`,
generation 2). TM2 then aged every database time by one hour before the
resume reached dispatch. Budget admission
(`lockRunBudgetRecoveryOwnerInTransaction`) required a live lease and
threw `Native recovery no longer owns provider dispatch`. The run failed
with `adapter_failed`, recovery stopped with
`native_continuation_requires_reconciliation` /
`preserve_without_replay_v1`, and the task needed Retry.

**Expected behavior**
The claim's own lapsed lease is taken back, and the resume is admitted
when the budget allows it.

**Steps to reproduce**
`node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts
--env local --adapters codex,claude --workloads A1,B5,C1 --faults
TM1,TM2,TM3 --time-age-hours 1 --concurrency 3 --tasks 6 --seed 9107`
(timing-dependent).

## What Changed

- `native-execution-lease.ts`: two new helpers. Both use the #20625
own-claim rule. Neither takes a lock out of order.
- `readOwnNativeRecoveryClaimLease` (read-only) returns `live`,
`lapsed_own_claim` or `lost`. It compares the lease against the database
clock. A lapsed lease counts as `lapsed_own_claim` only when all of
these hold:
    - the same lease owner;
    - this process's controller boot, pid and process start;
    - a non-null `recovery_state`;
- `nativeExecutionLeaseHeldCondition({ remote: false,
controllerGeneration })`, which applies the #20625 own-claim rule and
every #20474 untouched check;
    - no non-local environment lease on the run.
- `extendOwnNativeRecoveryClaimInTransaction` extends such a lease by
the local execution TTL, with one conditional UPDATE on the same
predicate.
- `budget-reservations.ts`:
- `lockRunBudgetRecoveryOwnerInTransaction` still refuses another owner,
and refuses a lapsed lease unless it is the run's own untouched claim.
It no longer locks the run row, so the documented coordinator → issue →
run order holds.
- The lease is extended only at the end of an admitted reservation
(`reserveRunBudget`) or an admitted dispatch recheck
(`recheckRunBudgetReservationInTransaction`). At those points the caller
already holds the run row lock. A refused admission, for example one
over budget, leaves the lease lapsed.
- `heartbeat-native-restart-suspension.test.ts`: the test "still settles
a recovery whose claim merely expired" encoded "expired means not the
owner". #20474 and #20625 changed that rule for an untouched own claim.
The test now has two cases:
- an expired claim that another actor recorded on (`awaiting_evidence`)
still settles with no provider session;
  - an untouched expired claim is dispatched.
- `doc/plans/2026-11-09-explicit-native-lease-claim-tokens.md`: Step 1
lists the new user of the rule.

## Verification

- New
`server/src/__tests__/native-recovery-dispatch-lease-reacquire.test.ts`:
- `reserveRunBudget` takes back the claim's own lapsed lease and admits
the resume. This fails before the change.
  - An over-budget resume is still refused, and its lease stays lapsed.
- Fenced variants each throw `Native recovery no longer owns provider
dispatch` and leave the row unchanged: a graceful-shutdown suspension,
an appended failure settlement, `awaiting_evidence`, `blocked`, a claim
from an earlier generation, another boot, a remote sandbox lease, and
another lease owner.
- At the dispatch gate (`recheckRunBudgetReservationInTransaction`), an
admitted own claim gets its lease extended, and a fenced claim is
refused with the row unchanged.
- Related suites pass (308 tests):
`native-execution-lease-reacquire.test.ts`,
`closed-native-receipt-accounting.test.ts`,
`heartbeat-native-predispatch-retry.test.ts`,
`run-dispatch/adapters/postgres.test.ts`,
`heartbeat-native-restart-suspension.test.ts` and
`heartbeat-restart-claim-lapse.test.ts`.
- `cd server && npx tsc --noEmit` passes.
- Live: one L7 wave (seed 9107) ran on the first commit of this branch
(`w-20261110T212307-s9107`). It logged 0 `Native recovery no longer owns
provider dispatch` refusals, against 1 in the previous wave. Codex B5
TM2 got past dispatch, but it still needed Retry for another reason:
after a `runner.drain` timeout, the scheduled same-run resume did not
send the continuation of #20645. That case is tracked separately. The
other five tasks passed.

## Risks

- This widens when a restart claim can take back its own lapsed lease,
and only at budget admission. It is local only. It needs the same owner,
boot and generation, and the row must show exactly that claim. A claim
that any other actor touched still fails as before.
- Budget hard-stop behaviour is unchanged. The lease is extended only
after budget admission passes.
- No schema change and no migration. Rows saved by earlier releases
carry the same history shape.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs). Extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-10 17:43:46 -06:00
DottaandClaude 30403781b9 fix(native): keep crediting proven unstarted restarts past the generation limit (#20647)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run survives a control-plane restart through restart
recovery, which resumes the interrupted provider turn
> - Restart recovery charges a provider attempt only when the provider
did work that could be lost, and it stops after three charged attempts
> - From controller generation 12 on, every restart charged an attempt
before that work rule was checked
> - In a busy restart wave, one long task sees 12 to 15 restarts, most
caused by other tasks. A restart that hit before the provider accepted a
turn was charged, the budget ran out, and a person had to click Retry
> - This pull request keeps the work rule past generation 12 for
restarts that a durable record proves did not start the provider, and
bounds a crash loop with a count of consecutive restarts without
provider progress
> - The benefit is that a long task survives many unrelated restarts
without a manual Retry, while a run that never makes progress still
stops

## Linked Issues or Issue Description

Refs #20625 (same restart recovery path). No public issue exists for
this case.

**What happened?**

`restartResumeKeepsAttempt` returned "charge" for every restart once the
run reached `NATIVE_PROGRESS_CREDIT_GENERATION_LIMIT` (12) controller
generations. It did this before it read the work-based outcome. In a
local restart chaos wave (L6, seed 9106, task t03, Claude B5 with RR1)
the run reached generation 13. The last restart interrupted a resume
whose runner started but whose provider had not accepted a turn. That
restart was charged at the last attempt, so the claim failed with
`execution_recovery_budget_exhausted` and
`native_restart_recovery_blocked`. The task needed a manual Retry.

**Expected behavior**

A restart that a durable record proves did not start provider work does
not spend an attempt, also past generation 12. A crash loop that never
makes provider progress still stops automatically. When it stops, the
task shows a Retry that works.

**Steps to reproduce**

Run a native task past controller generation 12. Make the last restart
hit after `native.process_start_requested` but before `turn.accepted`,
with the provider attempt at 3. The new integration test `does not
charge a restart past the generation limit whose resume never started
the provider` reproduces the failure before the fix.

**Paperclip version or commit**

`master` at ed5e7493bb.

## What Changed

- `restartResumeKeepsAttempt` now returns `{ keepsAttempt,
noProgressRestarts }`. A new helper judges the interrupted attempt as
`provider_not_started`, `provider_progressed` or `charged`, with the
same proofs as before.
- `noProgressRestarts` is 0 when the interrupted attempt is proven to
have made provider progress. Otherwise it is the previous claim's count
plus one. A clean-drain claim carries the previous count.
- Before generation 12 the decision does not change.
- From generation 12 on, a restart keeps the attempt only when the
interrupted attempt is proven `provider_not_started` and the count is
below the new `NATIVE_NO_PROGRESS_RESTART_LIMIT` (12). Provider work
(progress, or an accepted turn without progress) and unprovable cases
still charge.
- Each restart claim writes `noProgressRestarts` into its
`recovery_history` entry. No schema change: `recovery_history` is JSONB.
- Claims saved by earlier releases have no count. They are read as zero.
- Integration tests: the three generation-cap remote tests now check the
new bound. New tests cover the unstarted restart past the limit, the
consecutive-limit stop with a working Retry, the progress reset, an
accepted turn without progress, an unrecorded previous claim, and an
old-format claim.
- The durable-ownership plan (section F) records the rule.

**Why 12.** It equals the existing generation limit
(`PROVIDER_ATTEMPT_CREDIT_GENERATION_LIMIT`). The count includes
restarts before generation 12. Thus a run that never makes provider
progress is bounded exactly as before: the existing test `bounds a crash
loop that never starts the provider` passes unchanged. Only proven
progress resets the count. In the failing wave, the task had made
progress shortly before its last restart, so its count was 1.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
src/services/native-runtime/native-restart-recovery.test.ts
src/__tests__/heartbeat-native-restart-suspension.test.ts`
- Red before the fix: with only the new constant added to the old code,
the 9 new and changed tests fail. With the fix, they pass. The existing
crash-loop, progress and unstarted-resume tests pass without change.
- The consecutive-limit test checks that the stopped run is
`terminal_failure` / `native_restart_recovery_blocked` with no lease
owner and nothing scheduled, that `taskInterruptionPresentation` shows
`failed` with `canRetry: true`, and that `requestTaskRetry` accepts the
Retry.
- `cd server && npx tsc --noEmit -p .`
- Full local run of the three suites above: 322 passed, 14 skipped.
- Live: local restart chaos wave L6 (seed 9106, concurrency 3, `--env
local --adapters codex,claude --workloads A1,B5,C1 --faults RR1,RR2,RR3
--tasks 6`) at 7f7803ab0c: 6 of 6 tasks passed, and no task needed
Retry. Task t03 (Claude B5 RR1) reached controller generation 13 again.
The claim at generation 13 resumed the run (`resume_dead_runner`), and
the server log has no `execution_recovery_budget_exhausted`. The same
task on master was blocked at that claim and needed Retry.

## Risks

- A run that makes provider progress between restarts past generation 12
still charges each of those restarts. This is the same as before, by
design.
- An upgrade or a mid-run deploy reads old claims as count zero. A run
that was already in a no-progress loop can get up to 12 more
proven-unstarted restarts before the bound applies. It cannot be charged
where the old rule did not charge.
- A rollback ignores the new field and restores the old rule.
- Past generation 12 the claim now reads the event log, as it already
does before generation 12. This is not on the turn-start path.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, file edits, local test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-10 15:27:45 -06:00
DottaandClaude ed5e7493bb fix(native): take back a lapsed lease held under the run's own restart claim (#20625)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run holds an execution lease on its
`native_run_finalizations` row. The owner renews it every 5 minutes.
After a host sleep (or the restart-chaos time fault TM2), the stored
lease looks lapsed although the owner is alive.
> - #20474 lets the same owner take back a lapsed lease when no other
actor recorded anything on the row. One of the checks is
"`recovery_state` is null".
> - Restart recovery writes `recovery_state` (`resuming_session`,
`awaiting_runner_reattach` or `bootstrap_incomplete`) in the same update
that grants the lease. The executor keeps that value until a result is
accepted.
> - So a run that a restart resumed can never take back its own lapsed
lease. Renewal interrupts the turn, and failure settlement is refused
with `native_session_lease_lost`. The run fails with `adapter_failed`,
recovery holds it with `preserve_without_replay_v1`, and the user must
click Retry.
> - This pull request counts the claim's own record as untouched when
the row still shows exactly that claim and nothing recorded since.
> - The benefit is that a resumed run survives a later host sleep with
no Retry. Every real takeover still fences the late owner out.

## Linked Issues or Issue Description

Refs #20474

**What happened?**
The zero-retry restart-chaos matrix (local wave L7, seed 9107: TM1, TM2,
TM3 with 1 h aging) had 3 of 6 tasks that needed Retry. All three were
Codex runs. In each run, a TM3 kill made restart recovery resume the run
(`resume_dead_runner`, generation 2). The runner restored the
conversation, reported `provider_turn_lost_on_restore`, and the
same-conversation restart continuation started as designed. Then the TM2
fault aged every database time by one hour. The execution lease lapsed
while its owner was alive. The run then failed with
`native_session_lease_lost` (`adapter_failed`), and recovery stopped
with `native_continuation_requires_reconciliation` /
`preserve_without_replay_v1`. One run had already proposed its result
and completed its turn. The Retry click worked every time.

**Expected behavior**
A run whose lease lapsed only because time jumped, and that no other
actor claimed, continues with no Retry. This must also be true after a
restart resumed that run.

**Steps to reproduce**
`node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts
--env local --adapters codex,claude --workloads A1,B5,C1 --faults
TM1,TM2,TM3 --time-age-hours 1 --concurrency 3 --tasks 6 --seed 9107`

## What Changed

- `server/src/services/native-runtime/native-execution-lease.ts`: the
"untouched" part of `nativeExecutionLeaseHeldCondition` now also accepts
the run's own local restart-recovery claim. All of these must be true:
- the last `recovery_history` entry is that claim: same controller boot
and generation as the row;
- its disposition matches the current `recovery_state`
(`resume_dead_runner` → `resuming_session`, `reattach_existing_runner` →
`awaiting_runner_reattach`, `bootstrap_incomplete` →
`bootstrap_incomplete`);
  - `recovery_request_id` equals the request that claim recorded.
Any later actor appends a history entry (shutdown suspension, failure
settlement), moves the state (`awaiting_evidence`, `blocked`), or
replaces the owner, boot or generation. Each of these still fences the
late owner out. All other untouched checks are unchanged.
- `native-execution-lease-reacquire.test.ts`: new cases for each claim
kind, a requested claim, failure settlement under the claim, and eight
fenced variants.
- `doc/plans/2026-11-09-explicit-native-lease-claim-tokens.md`: Step 1
lists the new rule.

## Verification

- Red/green: `cd server && npx vitest run
src/services/native-runtime/native-execution-lease-reacquire.test.ts`.
Without the source change, 5 new cases fail (the three claim kinds, the
requested claim, and failure settlement). With it, all 37 pass.
- Related suites pass: `native-execution-lease.test.ts`,
`heartbeat-restart-claim-lapse.test.ts`,
`heartbeat-native-orphan-sweep-lease-race.test.ts`,
`native-runner-restart-lease-renewal.integration.test.ts`,
`native-runner-restart-preparation.integration.test.ts`.
- `cd server && npx tsc --noEmit` passes.
- Live: the same L7 wave (seed 9107, local) on this branch. The server
log has 0 `native_session_lease_lost` (the baseline had 3). The two
tasks whose cells failed this way in the baseline (codex C1 TM1, codex
C1 TM3) now pass with no Retry. Two other tasks needed Retry for
different causes, which this pull request does not change: codex B5 TM2
(a `runner.drain` timeout, then a same-run retry that may not continue a
turn lost on restore) and claude C1 TM2 (`ACPX session is not open` on
`turn.start` until the retry budget ran out). The L6 wave (seed 9106)
also ran on this branch: 5 of 6 passed. Its one failure is
`execution_recovery_budget_exhausted` and is not related to leases.

## Risks

- The rule widens when an owner may take back its own lapsed lease. It
applies only to local runs, before a result, with the same owner token,
attempt, boot, pid, process start and generation. A claim that any other
actor changed afterwards does not match.
- Rows saved by earlier releases carry the same history entry shape, so
a run claimed before a deploy is handled the same way. A deploy changes
the boot, so the new process never takes back the old process's lease.
- No schema change and no migration.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs). Extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-10 14:31:15 -06:00
DottaandClaude Opus 5.5 1b7ec13acb perf(native): default remote sandbox runs to a warm, reusable sandbox for measured harnesses (#20048)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents can run in remote sandboxes (Daytona), where each turn needs
the task's workspace and a runner inside the sandbox
> - With a default environment, every remote turn got a brand-new
sandbox: the whole workspace was uploaded again and a new runner
started, even for a follow-up comment
> - In a paperclip-sized workspace that made a follow-up take 30 s to
first token, against 8–10 s with a reusable warm sandbox
> - This pull request makes an agent that never chose a lifecycle run
warm on a sandbox for the measured harnesses (Codex, OpenCode, Claude),
so a follow-up keeps its sandbox, runner and workspace
> - The default applies only where warm can work: a sandbox provider
that keeps reusable leases, and a managed AI account that supports a
warm session. Everywhere else the run is per-turn, exactly as before
> - The benefit is fast remote follow-ups by default, without
re-uploading a large repository every turn

## Linked Issues or Issue Description

**What happened?**
On Daytona, with a fresh paperclip clone as the project workspace, a
follow-up comment took 30 s to its first token. The run log showed
`environment: fresh/created` on every follow-up. It spent 13.3 s in
`workspace_sync.transfer`, a full upload into the new sandbox, and 8–9 s
in `runner.session.resume`.

**Steps to reproduce**
1. Create a Daytona environment with the defaults: lifecycle inherited,
lease reuse off.
2. Run a Codex task in a large repository, then post a follow-up.
3. The follow-up gets a new sandbox, uploads the whole workspace again
and starts a new runner.

**Expected behavior**
A follow-up on the same task reuses the live sandbox, its runner and its
workspace (plan rule 2: no full re-upload per turn).

## What Changed

- **One rule for the default.** `defaultsRemoteSandboxRunToWarm`
(`native-runtime/native-warm-capability.ts`) decides the remote-sandbox
warm default for both the lease (environment view) and the runner
(native session policy), in one vocabulary (runtime backend plus ACPX
agent). It is applied only when no lifecycle was chosen anywhere:
admitted session, environment `runnerLifecycleMode`, or agent
`lifecycleMode`. It holds only when:
  1. the harness is measured (Codex, OpenCode, ACPX Claude);
2. the sandbox provider keeps reusable leases (Daytona, CreateOS and
boxd declare them; E2B, Modal and the built-in `fake` do not); and
3. with a managed AI account, that harness supports a managed warm
session (`supportsManagedNativeWarmSession`: Codex).
- **The harness resolves as the runtime resolves it.**
`resolveWarmProviderSelection` uses
`resolvePaperclipRunnerProviderProfile`, so an omitted `provider` is
Codex and ACPX without an agent is Claude, in both decisions. A config
the runtime would refuse gets no default.
- **Lease side** (`runner-environment-lifecycle.ts`,
`environment-run-orchestrator.ts`): for an eligible run only, the
orchestrator asks the provider before acquisition
(`providerSupportsReusableLeases` in `environment-runtime.ts`: the
acquisition rule, one registry lookup for a plugin provider, no
worker-ready wait). With support, the run-scoped view sets `reuseLease:
true` and forwards the warm idle window to the provider's auto-stop. The
shared environment is never rewritten. Every other run resolves exactly
as on master, without the probe.
- **Runner side** (`native-lifecycle.ts`, `runtime-selection.ts`): the
native policy applies the same rule to the acquired lease
(`reusableLeaseConfigured && effectiveCapabilities.reusableLeases`). If
acquisition downgrades the lease anyway, the run falls back to per-turn;
a default never produces
`runner_warm_environment_requires_reusable_lease`.
- **Idle window.** The remote default uses the window the environment
view forwards to the provider: an inheriting environment's
`runnerIdleTimeoutMs`, then the agent's `idleTimeoutMs`, then 5 minutes.
The sandbox execution target now carries the environment's window
(`runnerIdleTimeoutMs`, an optional field in `adapter-utils`'s execution
target) so the runner and the provider auto-stop agree. An explicit warm
choice and the local default are unchanged.
- Unchanged: explicit agent choices, environments that pin a lifecycle,
plugin-task environments, admitted sessions (`persistedPolicy`), and
local behavior.
- `tests/restart-chaos`: the vCPU budget check counts `--daytona-inherit
--lifecycle default` sandboxes as retained, because they now default to
warm.
- `doc/plans/2026-11-08-large-workspace-speed.md`: the measurements, the
ranked waits, and the gate.

## Verification

- Unit and integration tests, one per rule:
- Rule 1 (provider gate): `native-lifecycle.test.ts` (ephemeral lease →
per-turn), `runner-environment-lifecycle.test.ts` (no reusable leases →
environment unchanged, no warm window),
`environment-runtime-reusable-lease-probe.test.ts` (`fake`, an
undeclared plugin, a declared plugin whose worker does not verify the
reuse methods, and a plugin without a running worker read false),
`runtime-selection.test.ts` ("defaults an agent without a lifecycle on
...": reusable → warm/keep_running; reuse configured on an unsupported
provider → per_turn/destroy_after_turn; ephemeral → per_turn),
`environment-run-orchestrator.test.ts` (probe result decides
`reuseLease` and the forwarded window).
- Rule 2 (explicit configuration wins): explicit per-turn agent, pinned
environment, admitted session, configured `idleTimeoutMs`, managed
Claude per-turn, managed Codex warm; the orchestrator does not probe
when a lifecycle is explicit.
- Rule 3 (omitted provider): `{}` resolves to Codex in the environment
view; `runtime-selection.test.ts` "agrees with the environment view's
lease for ..." feeds the environment view's lease into selection for
Codex, a provider-less Codex agent, OpenCode, ACPX Claude, ACPX without
an agent, and Copilot (per-turn), and checks both decisions agree. A
plugin-task target stays out of the default.
- The two suites that failed on the previous head
(`runtime-selection.test.ts`,
`workspace-recovery-preparation-dispatch.test.ts`) pass locally.
Locally, 6 connector-skill cases in `runtime-selection.test.ts` fail
with `ENOENT .claude/skills/slack/SKILL.md` because this checkout is a
nested worktree; they do not touch this change and pass in CI.
- End-to-end check of this default on Daytona (before the gate was
added; Daytona keeps reusable leases, so the gate does not change this
path): a user-default environment (lifecycle inherited, lease reuse
unset, `--daytona-inherit`), a fresh paperclip clone, built from the
image's source revision with this change (branch measure/remote-warm).

  | | Before (per-turn sandbox) | After (this change) |
  |---|---|---|
  | Follow-up, Codex | 30.4 s | 7.6 s |
  | Follow-up, Claude | about 30 s | 11.0 s |
  | First follow-up after an idle K1, Codex | 45.9 s | 20.3 s |
  | First follow-up after an idle K1, Claude | 38.3 s | 18.3 s |

The run-log reuse records show the follow-ups reusing the workspace,
environment, runner and provider session, with no
`workspace_sync.transfer`. First turns still upload the workspace once
(14.8 s and 26.1 s). Every sandbox was cleaned up.
- Daytona restart chaos wave (K1/K2/K4 with a mid-run comment,
inheriting environment, agents with no lifecycle, runner image built
from this branch): wave `w-20261110T171438-s20048`, seed 20048, on
b7f6243d60 with image `paperclip-daytona-runner@sha256:76a33275…6fe29`
(built from this branch by the `image_only` runner workflow; runner
content id `ed503454…a479` matches every later commit here, which change
only server, adapter-utils types and tests). `--daytona-inherit
--lifecycle default --adapters codex,claude --workloads A1,C1 --triggers
time,comment --faults K1,K2,K4 --tasks 6 --concurrency 3`.
- **6/6 tasks pass** (R1–R6 and T), all 6 crossed a restart; cells:
Codex A1 K2, Codex A1 K4, Claude A1 K4, Claude C1 K1, Claude C1 K4,
Codex C1 K2 at the mid-run comment. **No task needed Retry** (R3 n/a on
every task; one Claude task recovered automatically from
`process_lost`). No open signatures, 0 speed breaches.
- The default applied: the R6 sweep found 6 retained reusable leases
(one per task, Daytona keeps them) and no orphan or leaked sandbox; the
environment was deleted cleanly.
- Timings (Daytona, first turns, n=3 each): TTFT p50 Codex 21.9 s,
Claude 31.0 s. Restart recovery p50/p95: Codex 10.5 s / 29.6 s (n=3),
Claude 8.8 s / 23.9 s (n=2). Warm follow-up TTFT has no clean sample in
this wave (both follow-ups were crossed by a fault), so the follow-up
comparison above (30.4 s → 7.6 s Codex, ~30 s → 11.0 s Claude) remains
the speed evidence; no separate per-turn chaos wave was run.
- An earlier attempt, `w-20261110T170357-s20048`, never started tasks:
the first sandbox from the new image stayed `creating` past the
harness's 300 s Codex qualification bound (Daytona building the new
snapshot). It left one sandbox that the harness reported after deleting
the environment; the rerun on the cached snapshot is the wave above.
- The idle-window fix (9a74116082) landed after the wave; it changes
only runs whose inheriting environment sets `runnerIdleTimeoutMs`, which
this wave's environment does not.

## Previous-release story

No persisted state changes shape. A session admitted per-turn before
this change keeps its saved policy (`persistedPolicy` wins) until a new
session starts. A run in flight at the cutover keeps its admitted
lifecycle for recovery (`admittedLifecycleMode` wins in the environment
view).

## Cost and opt-out

- **Cost:** after a turn, the sandbox and its warm runner stay up for
the warm idle window (the environment's `runnerIdleTimeoutMs`, else the
agent's `idleTimeoutMs`, else 5 minutes). Then the controller suspends
the runner, and the provider auto-stop follows the same window. One idle
window per active task.
- **Opt-out:** set the agent's lifecycle to "Turn by turn"
(`lifecycleMode: per_turn`), or set the environment's
`runnerLifecycleMode` to `per_turn`. Either wins over this default.

## Reuse scope, restart and saved work

- **One lineage per sandbox.** The Daytona plugin keys a reusable lease
by the agent plus the execution workspace (or issue). Another task never
gets a sandbox holding a different task's files, and subtasks still get
fresh workspaces from the project base, so they get their own sandbox.
The workspace is adopted only after its stamp matches the host's.
- **After a restart,** a warm sandbox is reattached or replaced through
the existing warm recovery and owner fencing. A sandbox with a live
owner is never claimed twice. The Daytona chaos wave above checks this.
- **No lost work.** Each turn's workspace copyback
(`workspace_finalize`) runs at the end of the turn, before any idle
suspend, as for per-turn runs.

## Risks

- **Cost:** an idle sandbox stays up for the warm idle window (5
minutes) after a turn, then suspends. Only on providers with reusable
leases.
- **Probe cost:** one plugin-registry lookup before acquisition, only
for runs the default can apply to. No wait for a worker: a provider
whose worker is not running yet reads as unsupported, so that run is
per-turn.
- **Known gap (pre-existing, not widened):** with an explicit agent warm
choice on an inheriting environment that sets `runnerIdleTimeoutMs`, the
provider gets the environment's window while the host uses the agent's,
as on master.
- **Restart behavior on warm remote sandboxes** uses the existing warm
recovery paths. The Daytona chaos wave above checks K1, K2 and K4.
- Sessions admitted per-turn before this change stay per-turn until a
new session starts.
## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file editing, test execution, Daytona measurements).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-10 11:49:14 -06:00
DottaandPaperclip bf0220d34a perf(workspaces): migrate with one archive and no readback (#20554)
## Thinking Path

> - Paperclip coordinates agents and keeps their workspace files.
> - A legacy workspace must move from a local volume to object storage.
> - The importer creates an archive, downloads and restores it, and
captures the source again.
> - Parent imports also download and recapture children that already
have committed snapshots.
> - This repeated work makes the one-time migration slow.
> - This change copies each source file once and trusts the upload
acknowledgement.
> - Existing source locks and atomic publication still own the
migration.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

The one-time migration of legacy local and Git workspaces to durable
snapshots.

**Current behavior**

Each workspace is captured twice and restored as a test during
migration. Parent imports also read back already imported child
snapshots. Capture hashes source files before writing them and scans
them again afterward.

**Proposed behavior**

Build one archive from one source-file read, encrypt it, upload it, and
publish its reference. Trust the upload acknowledgement. Reuse journaled
uploads after an interrupted process. Retain committed child snapshots
by database reference.

**Reason and benefit**

Remove repeated source reads, temporary copies, complete downloads, and
recursive child verification. The tests assert one capture, one upload,
and zero migration downloads.

**Breaking changes**

The archive format stays compatible. File hashes required by current
readers are computed inline during the copy. There is no preliminary
file-hashing pass. Existing readers can cold-restore the result.

Related work: #19817 and #20447.

## What Changed

- Add single-pass capture that fills fixed-length checksum fields in the
existing archive header after copying.
- Remove migration readback, test extraction, fresh-source recapture,
and Git integrity scans.
- Compute the encrypted archive checksum while writing it.
- Reuse acknowledged and journaled uploads without rebuilding or
downloading them.
- Retain already imported children through their database bindings.
Remove the recursive payload verification cache.
- Keep company checks, source and writer locks, recovery bindings,
journal identities, cancellation, and atomic publication.
- Bind archived repository intent paths to the operation without
reconstructing an archive to check them.
- Update the migration contract and the affected tests.

## Verification

- Node 24.22.2 and pnpm 9.15.4 on macOS arm64.
- At e876dbdc: focused capture, migration, lost-acknowledgement, and
frozen old-reader compatibility checks: 11 passed.
- Real source-file open counts prove one read per file. The single-pass
archive matches the existing archive bytes and manifest.
- Cold restore preserves staged and unstaged Git work, untracked files,
executable modes, and confined symlinks.
- At e876dbdc: full archive and cohort suites: 147 passed.
- At e876dbdc: server layered typecheck passed.
- At 58e5c229: four review fixes add Git source-root confinement,
archived lost-acknowledgement reuse, old-journal intent binding, and
wrapped-key readability. New focused regressions are included;
current-revision CI is pending.
- Post-review local runs were inconclusive: Git cases timed out in both
existing strict and single-pass modes, and DB tests stalled before
assertions. The unchanged earlier passes are not presented as
final-revision passes. Required CI will verify the final revision.
- Earlier full migration, snapshot storage, and nested-reference suites
passed before the compatibility refinement. Required CI supplies fresh
broad coverage for the final revision.
- Initial compatibility checks exposed the need to retain the existing
file-hash header. Inline checksums fixed that failure without another
source-file pass.
- Initial tests that required repeated verification were updated to the
requested copy-once contract. Upload-time database mutations still test
the retained business-state guards.

## Risks

- Migration now trusts the storage acknowledgement and cooperating
writer fences. It does not prove the uploaded bytes with a download or
detect later local edits with a second capture.
- Already imported child objects are trusted without another object
read. Their database bindings remain checked.
- Original source files remain available for recovery.
- This adds no automatic migration, server pin, or serving deployment.
Ordinary run capture and authenticated restore retain their existing
behavior.

## Model Used

OpenAI Codex, GPT-6 family, with code editing and terminal tools. The
exact runtime model revision and context window are not exposed in this
session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used, including the runtime details
available to this session
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
related work above
- [x] I have described the issue in-PR following the enhancement issue
template
- [x] I have not referenced internal or instance-local Paperclip issues
or links
- [x] My branch name describes the change and contains no internal
ticket ID
- [x] I have run relevant tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation
- [x] I have considered and documented risks
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open follow-ups
- [x] I will address reviewer comments before requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-10 09:13:33 -06:00
f4337bb5fb test(evals): land the chat catalog eval harness on master (#19614)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat-feature evals measure whether agents complete real chat
requests through the product
> - The chat catalog eval harness and its repairs live on the
`chat-eval/master-candidate` branch, not on master
> - Today's work made 139 of 242 chat evals pass in verified single-case
live runs. Each product fix landed on master as its own pull request
> - The eval-side changes (setups, graders, fault seams, evidence
capture) still exist only on the candidate
> - This pull request lands the harness on master. Master stays the
source of truth for product behavior
> - The benefit is one tree: evals run against master commits, and
product fixes reach the evals as soon as they merge

## Linked Issues or Issue Description

Refs #19441, Refs #19599, Refs #20270, Refs #20334, Refs #20342, Refs
#20344, Refs #20354, Refs #20360, Refs #20373, Refs #20382, Refs #20409

**What existing behavior does this improve?**
The chat catalog eval harness and the product it evaluates live on a
long-running branch outside master. Eval fixes do not reach master, so
master-based eval runs use old setups and graders.

**Subsystem affected**
`tests/runner-e2e` chat catalog harness, plus a small set of product
changes that exist only on the candidate.

**Proposed change**
Land the harness on master. Every product fix from this eval work
already merged to master as its own pull request. Conflicts keep
master's product behavior.

## What Changed

- Most of the diff (808 of 851 files) is under `tests/runner-e2e/`:
chat-catalog domains, scenarios, descriptors, the TestOps manifest,
private evidence tooling, and their tests.
- Graders now check saved state, setup literals and rendered or clicked
links. Many keyword and regex checks on free-form replies are removed.
Each changed grader has a bumped `oracleVersion` and calibration tests
(a valid reply, a reworded valid reply, and broken variants that must
fail).
- The shared link helper `hasLink` no longer treats a list link (for
example `/COM/issues?view=mine`) as a link to every task without a URL
key.
- Setup fixes build real state through supported product routes: routine
history, agent grants through the board grant route, budget holds with a
real cap, and a storage-layer archive fault for
`inbox-search.archive-partial-failure`.
- Browser boundary journeys (sign-in, chat-disabled, onboarding) now
write the standard pass evidence files.
- Retargeted cases, approved by the maintainer:
`agent-chat.chat-disabled` (Conference Room gate),
`agents.restore-instructions` (no saved instruction history),
`access.manage-membership` (member permission edits stay board-only).
- The private evidence trusted files (`manual-private-handoff.ts`,
`manual-private-retention.ts`, `manual-private-report-assets.ts`) are
byte-identical to master.
- Product files outside `tests/` (43 files, +1,328 −49) carry
candidate-only changes that master does not yet have. Most are tests.
The main code changes are in `request-human-input.ts`,
`heartbeat-run-summary.ts`, `native-completion-feedback.ts`,
`native-run-finalizer.ts`, `openapi.ts`, `secrets.ts` and
`IssueChatLongThreadPerf.tsx`.

## Verification

- `pnpm vitest run chat-catalog-domains chat-catalog-contract.test.ts
chat-catalog-consistency.test.ts unit-launcher.test.mjs
--testTimeout=180000` in `tests/runner-e2e`: 269 files pass, 3,550 tests
pass.
- `node --test chat-full-catalog-testops-manifest.test.mjs`: 10 pass.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`:
no change.
- `tsc` for `server` and `tests/runner-e2e`: clean.
- Live single-case verification: 139 of 242 cases passed on paid runs,
and a person read the saved evidence for each. These are single-case
runs, not one full-cohort run.

## Risks

- Large diff. Most of it is test-only, but the 43 non-test files change
product code paths. Review those files first.
- Some graders are now less strict on reply wording. A person must read
replies for meaning on those cases.
- 42 cases need disposable outside accounts (Notion, Slack, GitHub,
Daytona and others) and stay blocked. `access.invite-teammate` stays
failing until local agent isolation lands, because the runner's durable
state keeps a live invite token.
- The paid workflow compares the trusted private-evidence files with
master. They are unchanged, so this risk is low.

## Model Used

Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code
execution, in Claude Code. Subagents used the same model.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-10 08:02:11 -06:00
DottaandClaude dc9971bdb8 fix(native): keep another server's orphan sweep off a lease its owner took back (#20508)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run holds an executor lease on its
`native_run_finalizations` row. #20474 lets the same owner take back its
own lease when the lease lapsed only because the host slept.
> - That take-back is safe only if every other actor that acts on a
lapsed lease records its action under the coordinator row lock first.
The plan `doc/plans/2026-11-09-explicit-native-lease-claim-tokens.md`
lists the actors that do not. Gap 2 is another server's orphan sweep.
> - The sweep reads the lease as lapsed. Later it writes the
ownership-unverified marker on the run, without the coordinator lock and
without a new check of the lease. If the owner wakes and takes its lease
back between the read and the write, a healthy run gets the marker.
> - The board then shows "automatic recovery is blocked" on a run that
works. After the next host sleep, the owner cannot take its lease back,
so the turn is interrupted and the user must click Retry. The owner's
rule is that a task interrupted by sleep must resume by itself.
> - This pull request makes the sweep write its marker under the
coordinator row lock, and only while no executor lease on the run is
live.
> - The benefit is that exactly one side wins: the sweep marks a run
whose owner did not come back, and it leaves alone a run whose owner
took its lease back.

## Linked Issues or Issue Description

Refs #20474

**What happened?**
Two servers share one database. Server A's host sleeps, so the executor
lease of its native run lapses. Server B's orphan sweep reads the lapsed
lease. Server A wakes and takes its lease back (#20474). Server B then
writes `native_execution_ownership_unverified` on the run that A still
owns.

**What did you expect?**
Server B writes the marker only if the lease is still lapsed when it
writes. One side wins.

**Steps to reproduce**
`server/src/__tests__/heartbeat-native-orphan-sweep-lease-race.test.ts`
holds the owner's take-back open, runs the real
`heartbeatService(db).reapOrphanedRuns()` until its write waits on a
lock, then commits the take-back. Before this change the run ends with
`errorCode: native_execution_ownership_unverified` and the owner's next
take-back fails with `native_session_lease_lost`.

## What Changed

- `setRunStatusFromLive`
(`server/src/services/heartbeat/run-lifecycle.ts`) has a new option
`onlyWhileNativeLeaseLapsed`. It locks the coordinator row first. This
is the same order that `lockNativeExecutionLeaseForReacquire` uses. The
UPDATE then requires that no executor lease on the run is live.
- `markNativeOwnershipUnverified`
(`server/src/services/heartbeat/recovery.ts`) takes `leaseLapsed`. The
orphan sweep in `recoverActiveExecutions` sets it when it read the lease
as lapsed or unset. Other callers do not change. They keep the same call
shape.
- A new regression test, red before the fix:
`server/src/__tests__/heartbeat-native-orphan-sweep-lease-race.test.ts`.
It also checks that the sweep still marks a lapsed run whose owner did
not come back, and that the late owner stays fenced out.
- `doc/plans/2026-11-09-explicit-native-lease-claim-tokens.md`: gap 2 is
marked closed without the epoch design.

## Verification

- `cd server && pnpm exec vitest run
src/__tests__/heartbeat-native-orphan-sweep-lease-race.test.ts` — red on
`origin/master` (the run keeps the marker), green with this change.
- `cd server && pnpm exec vitest run
src/__tests__/heartbeat-native-orphan-sweep-lease-race.test.ts
src/services/heartbeat/recovery.test.ts
src/services/native-runtime/native-execution-lease-reacquire.test.ts
src/__tests__/heartbeat-status-preserving-telemetry.test.ts` — 129
passed.
- `cd server && pnpm exec tsc --noEmit -p tsconfig.json` — clean.
- Restart-chaos time-fault wave on this branch: see the wave comment
below.

## Risks

- Low risk. Only the orphan sweep's marker write changes, and only for a
run whose lease it read as lapsed. The sweep now takes the coordinator
row lock before the run row. The owner's take-back uses the same order,
so the two cannot deadlock.
- If the owner took the lease back, the sweep now writes nothing and
appends no event. The next sweep sees a live lease and defers, as
before.
- The other gaps in the plan (workspace finalization, shared-workspace
holder detection, board copyback confirmation) do not change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-10 03:57:20 -06:00
DottaandClaude 1402e92d1f fix(native): take back an executor lease that lapsed only because the host slept (#20474)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run holds an execution lease on its
`native_run_finalizations` row. The lease lasts 20 minutes, and the
owner renews it every 5 minutes.
> - When a laptop sleeps for longer than the lease, the wall clock and
the database clock move past `lease_expires_at`. The server's monotonic
timers barely move.
> - After the wake, the next renewal finds its own lease expired. It
interrupts a healthy turn with `native_session_lease_lost`, and the user
must click Retry. Nobody else took the lease.
> - Several background consumers act on an expired lease without
changing its owner. So expiry alone cannot simply be ignored for the
owner.
> - This pull request makes takeover explicit. The same owner can take
back its own lapsed lease only when no other actor has recorded anything
on the row or the run.
> - The benefit is that a run interrupted only by host sleep continues
with no Retry. A real takeover still fences the late owner out.

## Linked Issues or Issue Description

**What happened?**
The restart-chaos time fault TM2 stops the server and runner tree and
ages every database time by one hour, as a stand-in for a laptop that
slept. After the thaw, the next 5-minute renewal of the native execution
lease found the lease expired. It interrupted the turn
(`turn.interrupted` at 300 s, `native_session_lease_lost`). The failure
settlement was then refused with the same code, and the UI showed
"Couldn't finish this attempt" with Retry. The owner, attempt, boot and
pid were unchanged. Evidence: wave `w-20261110T051535-s304` t01 (TM2, 1
h) failed R3. t02 in the same wave (TM1, a 113 s server-only freeze with
no clock jump) passed, so the failure follows the clock jump and not the
pause.

**Is this a harness artifact?**
No. TM2 ages the stored rows, and a real sleep moves the clock forward.
Both produce the same comparison: a stored `lease_expires_at` is earlier
than `clock_timestamp()` after the wake. The server's renewal timer and
local deadline use the monotonic clock. On macOS and Linux that clock
does not count suspended time, so they do not notice the sleep in either
case. The first database check after the wake is the renewal predicate
`lease_expires_at > clock_timestamp()`, and it fails in both cases.

**Expected behavior**
A run whose lease lapsed only because time jumped, and that no other
actor claimed, keeps running and finishes without Retry.

**Steps to reproduce**
`node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts
--env local --adapters codex --workloads B5 --faults TM2
--time-age-hours 1 --concurrency 1 --tasks 2 --seed 304` with the
harness from #20458.

Refs #20458, #20500

## What Changed

- `native-execution-lease.ts`: add `nativeExecutionLeaseHeldCondition`
and `lockNativeExecutionLeaseForReacquire`. A lease is held when it is
live. For a local run, it is also held when it lapsed and is untouched.
"Untouched" means all of these are true:
  - the claimed generation matches;
  - the phase is `observed` and no result is accepted;
- `recovery_state`, `recovery_request_id`, `failure_code` and
`next_attempt_at` are null;
  - `controller_host_hash` is not the board's attested-stopped stamp;
- the run is still `running`, without the ownership-unverified error
code and without a `nativeStartupCleanup` marker.
- The caller's WHERE already matches the owner token, attempt and
controller boot/pid/start.
- The lock helper locks the coordinator row. If the lease lapsed, it
also takes a share lock on the run row. The predicate then reads the
latest committed run, and an in-flight run write cannot slip past it.
- Renewal (`startNativeSessionExecutionLeaseRenewal` /
`renewNativeSessionExecutionLease`) takes back a lapsed lease when the
executor passes its claimed generation. Remote targets never do. A
deadlock (40P01) on that rare path retries at the next tick, as a
statement timeout already does.
- `native-session-executor.ts`: the executor passes its generation to
renewal. Failure settlement and the final release use the same lock and
condition. After a result is accepted, a lapsed lease can only be
relinquished (set to null), never renewed, because workspace
finalization may act on it.
- New `native-execution-lease-reacquire.test.ts` (embedded Postgres, 24
cases).
- `doc/plans/2026-11-09-explicit-native-lease-claim-tokens.md`: step 2,
which adds explicit claim tokens for the consumers this PR does not yet
move.

Consumer inventory (readers of
`native_run_finalizations.lease_expires_at` that can act on a lapsed
lease):
- Restart recovery and takeover (`claimNativeRestartRecoveries`): claims
first. It writes `recovery_state`, `lease_owner` or `controller_*` under
the coordinator lock.
- Finalizer and cleanup claims: claim first (`lease_owner`).
- Failed-run owner release, environment settlement, observed-workspace
preservation, retained-session cleanup: require a terminal run status,
`committed`, or a recovery state. The rule refuses all of these.
- Finalization reconciler, live recovery: require `retryable_failure`, a
field the rule checks.
- Board controller-stopped confirmation: writes `controller_host_hash`,
which the rule checks.
- Startup cleanup promotion: writes the run's `nativeStartupCleanup`
marker, which the rule checks. It also applies only to runner-task
leases.
- Another server's orphan sweep: writes the run's ownership-unverified
code. The rule checks it when the marker lands first. The reverse order
is deferred (see Risks).
- Workspace finalization copyback: acts on a lapsed post-result lease
without a coordinator write. Post-result leases are therefore never
renewed (deferred to step 2).
- Accepted-result and token lease settlement, checkpoints, cancellation
delivery, admission and the other readers: act on live leases only, or
are read-only.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-execution-lease-reacquire.test.ts`
passes (24/24) on this branch. With the source files from
`origin/master`, 5 fail, including "lets the same owner take back a
lease that lapsed untouched" and "renews from the shared timer after the
sleep instead of interrupting the turn".
- (a) The owner takes back its lapsed lease when no other actor claimed
it. Also covered: the shared 5-minute timer, rows saved by the previous
release, and the post-result relinquish.
- (b) 14 cases where a consumer claimed first fence the owner out and
leave the row unchanged. A release over a recorded recovery claim is
refused. A different boot (a deploy mid-run), a different generation, a
missing generation and a remote target are refused.
- (c) Races have exactly one winner: a claimant and the owner lock the
coordinator in each order, an in-flight run status write races the
owner, and 8 simultaneous starts each have one winner.
- Existing suites pass: `native-execution-lease.test.ts`,
`native-session-executor.test.ts`, `native-runner-restart-recovery`,
`native-runner-restart-lease-renewal`,
`native-runner-restart-preparation` and
`native-runner-restart-checkpoint-cost` integration tests.
- Live proof: `B5 × TM2 --time-age-hours 1 --tasks 2 --seed 304` with
`--server-from` this branch. Wave `w-20261110T083444-s304` passed 2/2,
with no Retry and no `native_session_lease_lost`.
- Both runs ran past the 300 s renewal after the one-hour aging, at the
exact point where `origin/master` fails. t01 ran 310 s and t02 ran 319
s. Each was a single run that succeeded.
- The harness used the aging exclusion from #20500 (merged): process
start identity is not aged. Without that exclusion, TM2 also moves
`controller_process_started_at`. The live owner then fails its identity
check, which a real sleep never causes. An earlier wave on the unpatched
harness, `w-20261110T072106-s304`, failed 0/2 for that reason only.

## Risks

- Previous release: no persisted shape changes and no migration. Rows
saved by the previous release have the same columns, and a null
`controller_host_hash` is accepted. A deploy that lands mid-run cannot
take back the old server's lease, because the boot and pid differ.
- On the normal path, a live lease takes the same single coordinator
lock as before. The run share lock is taken only after a lapse, and
nothing is added before `turn.start`.
- Deferred to step 2 (see the plan):
- The orphan sweep of another server process can read the row before the
owner takes the lease back and write after. That run would carry an
ownership-unverified error while it is live.
- Shared-workspace holder detection reads a silent run with lapsed
leases as gone.
  - Post-result leases are not renewed.
  - Remote leases are unchanged.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code
with tool use (shell, file editing, sub-agent code audit).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-10 02:47:53 -06:00
DottaandPaperclip 0934b834c4 perf(workspaces): release retained cohort parent scratch early (#20447)
## Thinking Path

> - Paperclip manages agent work and preserves its workspace files.
> - Online migration captures local files and publishes remote workspace
authority.
> - Each capture verifies encrypted readback, restores the archive and
compares the current source.
> - A cohort kept every completed parent payload until the whole cohort
published.
> - A larger cohort could retain many complete copies while later
members still worked.
> - This pull request releases only finished parent payloads before a
capture worker starts another member.
> - Nested-child proof, final fresh checks and atomic publication keep
their existing owners.

## Linked Issues or Issue Description

Refs: #20418, #20437, #20441.

**What happened?**

A cohort retained each member's capture archive, verified archive,
comparison archive and restored tree until atomic publication. The
capture worker limit did not limit that retained parent data.

**Expected behavior**

Release completed parent payloads once their consumers finish. Keep
files that a later publication check still needs.

**Steps to reproduce**

Run the added held-peer regression against the previous owner. It
completes one parent, holds a later member and finds all four completed
parent paths still present.

## What Changed

- Release three named parent archives and the restored tree after a
successful online cohort capture, before that worker takes another
member.
- Keep parent scratch roots and all nested-child verification areas.
Keep every final fresh child and publication check.
- Retain payloads for archived repository intents. Use the existing
writable-tree cleanup for read-only restored directories.
- Add real filesystem regressions for early release, retained child
proof, late child mutation and the final archived-intent path check.
Update the source-fence guide.

This branch incorporates the final #20441 owner and both distinct
saved-relative-path regressions. Its scratch-specific production change
is nine added lines and one removed line. Single imports and saved
journal/archive formats stay unchanged.

## Verification

- Canonical Node24 preflight passed. The Runner build came from the
content cache. After the current-master merge, the preflight checked
1,674 inputs and 1,817 outputs and found unchanged build inputs. Plugin
SDK prerequisites passed.
- Genuine RED on the unchanged #20441 owner: one held-peer case failed
at four retained payload paths; the archived-intent control passed. The
run took21.14s, with221 unrelated cases intentionally unselected. The
first sandbox-restricted setup attempt hit DNS errors and was stopped;
the normal network-qualified preflight then passed.
- Initial treatment: four selected PostgreSQL/filesystem cases passed
in38.61s, with219 intentionally unselected. The held peer observes
released parent payloads and retained child areas. A later child
mutation still refuses the whole cohort and publishes no authority or
activity.
- The archived-intent control observes all parent payloads at the actual
final saved-relative-path check. The changed path refuses publication
and keeps task/repository state intact. This exercises its actual
singleton owner; archived-only sources do not enter the online cohort
API.
- After a normal merge of current master and the final #20441 test, the
five affected cases and server TypeScript passed. No full local matrix
was repeated. Initial CI failed the inherited recovered-upload assertion
because the new retention test leaked its archive and filesystem spies.
That case passed alone but failed in the adjacent three-case sequence
(two pass, one fail); scoped `finally` cleanup made the same sequence
pass three of three in17.15s, with191 unrelated cases intentionally
unselected. No upstream assertion or production code changed. Server
TypeScript passed again. All required
[CI](https://github.com/paperclipai/paperclip/actions/runs/42746586511)
passed on final normal-merge head
`604c0f67bf7282ecd3f6e5e5da1c9d8637ef70cc` (56 success, four skipped and
one neutral; no pending or failing checks). The previously failing
server4/12 passed in10m23s. Fresh Greptile is5/5 on that exact head with
no open threads. The merge resolves the duplicate #20441 squash
insertion; both tested owner and test files are byte-for-byte identical
to repair head `b5c8eed82b6a676985e1a8af785e5df2ae3be2ce`. The prior
head received
[Greptile5/5](https://github.com/paperclipai/paperclip/pull/20447#issuecomment-6757435885);
its archived-cohort coverage warning was [answered using the actual
public admission
boundary](https://github.com/paperclipai/paperclip/pull/20447#discussion_r4520641017).
- `git diff --check` passed. No hosted cleanup, migration, provider
action or deployment was performed.

```sh
node scripts/agent-preflight.mjs --skip-rust-debug
pnpm --filter @paperclipai/plugin-sdk ensure-build-deps
node --import ./cli/node_modules/tsx/dist/loader.mjs packages/db/src/test-postgres-command.ts pnpm exec vitest run server/src/services/remote-workspace-cohort-migration.test.ts server/src/services/remote-workspace-migration.test.ts -t 'releases completed parent payload|retains archived intent payload|shares the merged child verifier|rechecks the current archived repository intent path at publication'
pnpm --filter @paperclipai/server exec tsc --noEmit
```

## Risks

- This reduces retained parent data. It does not establish a total disk
bound. Active capture/encryption/readback files and retained child areas
still need measured capacity.
- Cleanup failure uses the existing worker abort and peer-join path
before publication. Original source files, uploaded objects and journals
remain intact.
- Archived repository intents still need their restored payload at
publication. Those payloads stay retained. No capture limit, budget or
integrity guard changes.
- Saved operations and archive identities carry forward unchanged.
Resume still uses the existing full source and authenticated object
checks.

## Model Used

OpenAI GPT-6 (Codex), with tool use and code execution. The runtime does
not expose a more specific model ID or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-09 21:16:53 -06:00
DottaandClaude 1b52f359dd test(restart-chaos): add config fault waves (CF1-CF3) and the CF4 design (#20446)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart chaos harness (`tests/restart-chaos/`) starts an
isolated control plane, runs real agent tasks and restarts the server
during them
> - Every wave starts from a new instance with valid default settings
and one valid provider
> - On 2026-11-09 production failed on state that a new instance never
has: a setup command that named a renamed script (#20269), and an
account default that did not exist (#20361)
> - This pull request adds a "config fault" family (CF1–CF3): the
harness breaks one saved reference on a task's own agent or project,
restarts the server while the task is parked, repairs the setting and
grades the result
> - The benefit is that the harness now catches "stale saved setting"
failures, including a run that retries forever or a task that stays
Queued with no explanation

## Linked Issues or Issue Description

Refs #20269
Refs #20361

**What existing behavior does this improve?**
The restart chaos harness only tests new instances with valid settings.
It cannot see failures that come from saved settings that point at a
missing object.

**Subsystem affected**
`tests/restart-chaos/` (test harness only). No product code changes.

**Current behavior**
No wave breaks a saved setting. The CF2 failure (#20361) and the CF1
failure (#20269) were found in production, not by the harness.

**Proposed behavior**
`config-wave.ts` runs CF1–CF3. The oracle is not "the task completes". A
task passes when all of these are true:

- it completes, or it rests on a message that names the broken setting
on the task page within a bound;
- no run retries unchanged configuration forever;
- no task stays Queued with no owner;
- the rest survives a K1 or K2 restart;
- after repair and the offered action (connection card, Retry, or agent
setup Retry), the task completes.

## What Changed

- `tests/restart-chaos/config-faults.ts` (new, pure): the catalog of
broken references, the seeded plan, `parkState`, the task-level message
check and `gradeConfigTask`. CF1 is a missing setup script. CF2 is a
missing personal OpenRouter default. CF3 is a seeded walk over a deleted
secret, a missing workspace path, a missing base ref, an unknown model,
plus CF1 and CF2. `CONFIG_REFERENCES_NOT_BROKEN` lists the settings CF3
cannot break, and why.
- `tests/restart-chaos/config-wave.ts` (new): the wave runner. Each task
gets its own agent, credential secret and project. The runner breaks the
setting through board routes, launches an A1 brief task, polls, fires K1
or K2 while the task is parked, repairs the setting, uses the offered
action and grades the task. It reuses the existing supervisor, seed and
A1 oracle. Waves are limited to 4 tasks and a concurrency of 2. The
instance is deleted at the end.
- `tests/restart-chaos/config-faults.test.ts` (new): unit tests for the
plan, the arguments, `parkState` and the grader. These include the CF2
bug signature and the CF1 "message only on the run page" signature.
- `tests/restart-chaos/README.md`: a new "Config faults" section and a
row in the pieces table.
- `doc/plans/2026-11-09-chaos-config-and-aged-state-faults.md`: the CF4
design (an instance aged by an older release, built on the existing K14
`--upgrade-from` machinery). CF4 is not built in this pull request.
- There are no edits to `main.ts`, `types.ts`, `workloads.ts`,
`invariants.ts` or `grader.ts`. Other agents work on those files in
parallel.

## Verification

- `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`:
24 files, 291 tests pass (before the last unit test was added). The
config-fault suite has 13 tests, and all pass.
- `node scripts/typecheck-test-projects.mjs tests/restart-chaos`:
passes.
- Live waves (local, isolated instance, one wave at a time):
  - **CF2 red before #20361** (base `8004ad069e`, before #20361 merged):
- Waves `w-20261110T000602-s201-cf` and `w-20261110T004818-s202-cf` fail
with
`CF2/ai_default:bounded_rest+actionable_message+lifecycle_retrying_unchanged_config+queued_without_live_owner`.
- The user sees `Queued`, a `resume` user action, and "Execution
configuration check failed. Paperclip could not complete this step. It
will retry automatically." There is no card.
- A debug run with logging added to the lifecycle failure log (not
committed) showed that the underlying error is
`ai_connection_default_missing`. Verification retried it every 60 s,
more than 20 times.
- After the harness connects an account that is installed for the agent,
the task completes 35 s later on its own.
- The CF2 run after #20361, and the CF1 and CF3 waves on current master,
are added below as they finish.

## Risks

- Low risk. The change is test harness code and docs only.
- A wave costs provider tokens only when the repaired task runs (one A1
brief).
- The CF2 repair creates a real OpenRouter account in the isolated
instance. The server validates the key live.
- The oracle reads the task page through the API (interruption,
execution status, cards, agent setup status and notices). It does not
render the board UI. A message that appears only in the run's own error
text is graded `message_only_in_run_detail`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, file edits, subagents). Standard context window.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-09 21:10:42 -06:00
DottaandPaperclip 07f6e8ff75 perf(workspaces): group same-root cohorts with bounded workers (#20437)
## Thinking Path

> - Paperclip preserves agent work when its control-plane server is
replaced.
> - Online migration imports local workspaces into authenticated
snapshots.
> - Several workspace records can point to one physical directory.
> - Shared child verification avoids repeated work inside one held
cohort fence.
> - The eight-member limit splits 30 identical-root aliases into four
cohorts and repeats the child passes.
> - This pull request admits up to 32 members only for one canonical
physical root, while keeping each work stage bounded to eight peers.
> - The benefit is fewer repeated child checks while capture and
publication guards stay in place.

## Linked Issues or Issue Description

Refs: #19873, #19882, #20418, #20424, #20429. Related PR search found
shared verification and restart fixes, but no larger same-root cohort
owner. The shared verifier and archived-source changes are merged into
this branch from current master.

**Current behavior**

Thirty fresh aliases of one physical root require cohorts of 8, 8, 8 and
6. Each cohort performs an initial and final local child-validation
pass. The same children are checked eight times in total.

**Proposed behavior**

Admit one cohort with nine to 32 members only when every canonical
resolved physical root is exactly equal. Keep the existing complete
source-scope equality and admission guards. Keep mixed-leaf cohorts
limited to eight members. Bound source reads, reservations, recovery
checks, final readers, replay and cleanup to eight joined peers. Capture
concurrency stays at most eight.

**Reason and benefit**

For N distinct canonical covered children, the 30-alias case changes 8N
local child checks to 2N, a 75% reduction in repeated child validation.
If N is 68, that is 544 checks versus 136. Thirty parent captures,
uploads and restores remain required. No total wall-clock improvement is
claimed.

**Breaking changes**

Saved cohort fields and exact membership remain unchanged. Older tools
that enforce the eight-member bound refuse larger cohorts. Use the newer
tool to resume them; never split or regroup their saved membership.

## What Changed

- Raise the canonical service, saved-cohort and maintenance-request
member bounds to 32. Keep the capture-worker bound at eight and
single-workspace modes at one.
- Require one exact canonical physical root for more than eight members.
Keep the existing full source-scope comparison, leaf rule and P/I/L/F
guards.
- Use one private stage mapper with batches of at most eight. Join every
started peer before an error can leave the stage. Stop future batches
after a failure or cancellation.
- Preserve the existing cancellation and pending-snapshot ordering for
stages of at most eight members. Check cancellation around extended
batches.
- Keep all saved cohort, member, operation and snapshot identities,
proof refresh and atomic publication checks.
- Bound cleanup without skipping later scratch cleanup after an earlier
cleanup error.
- Extend the existing PostgreSQL fixture with ten real aliases,
partial-upload resume, cold restoration, replay and mixed-root refusal.
Extend the request boundary test.
- Document old-tool refusal, work counts and the retained-scratch
capacity bound.

## Verification

- Canonical Node24 worktree preflight and Plugin SDK prerequisites
passed. After merging current master, preflight passed again with
unchanged cached build inputs.
- Genuine RED on the old bound: both real ten-alias import/resume cases
and the 32-member request reject the eight-member limit (3 failed; 47
not selected; 18.23 seconds).
- First focused GREEN: 4 passed in 24.20 seconds. The fresh ten-alias
case performs exactly four local child captures for two children: one
initial and one final pass. Actual reader, transaction and upload maxima
are eight. All ten archives cold-restore after the original source is
parked; replay retains the same cohort.
- The partial-upload case keeps all ten saved operation IDs and the
existing snapshot IDs. More than eight members across different
canonical leaf roots refuse before any journal or storage work. The
request parser accepts 32 members and eight workers, and refuses 33
members or nine workers.
- The first expanded corpus had 55 passes and one real compatibility
failure: a new eager cancellation check skipped the old pending snapshot
boundary. The mapper now retains existing cancellation ordering for at
most eight peers. That failure remains recorded as a nonpass.
- Final sources include current master at `18697b8d21`, with the
verifier, archived-source and UI repairs. The selected four-file corpus
passes 47 cases in 78.88 seconds, with 191 intentionally not selected.
The full maintenance suite separately passes 23 cases in 10.21 seconds.
Nine cases overlap; the final union is 61 distinct passing checks.
- Final coverage includes all 27 cohort cases, all 23 maintenance cases,
all six actual old-format compatibility cases and five archived
prepared-source controls. Existing source mutation, finalizer
contention, actual fence loss, cancellation peer joins, journal
compare-and-swap and outer-commit rollback controls pass.
- Final server TypeScript, module boundaries and whitespace checks pass.
Only the resource-risk documentation changed after that test run;
runtime and fixture bytes stayed unchanged.
- Commands: `pnpm exec vitest run
server/src/services/remote-workspace-cohort-migration.test.ts
server/src/maintenance/online-workspace-migration.test.ts
server/src/__tests__/online-workspace-migration-compatibility.test.ts`;
archived controls use the existing `remote-workspace-migration.test.ts`
cases named `unselected archived recipe` and `archived prepared source`;
`pnpm --filter @paperclipai/server exec tsc --noEmit`; `node
scripts/check-module-boundaries.mjs`.
- Initial source review was 4/5 and requested a control for the extended
reader-batch stop boundary. The test-only follow-up
`ca80ae9cc8791a72dab632090488093c5d0916f3` adds actual ten-member
failure and cancellation cases. They hold a first-batch peer, verify
that the source fence stays held, and prove members nine and ten never
start preflight. Both pass in 10.87 seconds with no new journal,
snapshot or upload effects. Runtime source hashes are unchanged; the
local union is now 63 distinct checks.
- The first stop-test attempt observed the two controlled readers before
all SQL-backed peers had entered the inspection callback. Its assertion
was too early. The corrected test checks the held first-batch readers at
that point, then requires all eight to join before completion. Both
cases pass; the earlier attempt remains a nonpass.
- Final head `ca80ae9cc8791a72dab632090488093c5d0916f3`: [required
CI](https://github.com/paperclipai/paperclip/actions/runs/42737676664)
and [aggregate
verify](https://github.com/paperclipai/paperclip/actions/runs/42737676664/job/129575813765)
passed. Full rollup: 56 successes, four intentional skips and one
advisory neutral result. Classic Snyk passed.
[Greptile](https://github.com/paperclipai/paperclip/pull/20437#issuecomment-6756419093)
and
[Ponytail](https://github.com/paperclipai/paperclip/pull/20437#issuecomment-6756388693)
both reviewed that exact head at 5/5. The one review finding is
addressed and resolved; no open findings remain. Older partial CI and
the initial 4/5 review remain nonfinal evidence. Hosted acceptance and
measured wall-clock savings remain pending.

## Risks

- Completed parent captures retain their scratch until the atomic
publication completes. A 32-member cohort can retain up to four times
the parent scratch of an eight-member cohort. Check actual parent
capture, verification and restore sizes against current target
filesystem capacity before selecting a larger cohort. Keep the existing
resource guards.
- Larger member arrays must not raise execution fanout. Every
reader/reservation/final stage is bounded to eight and joins current
peers before failure. Capture workers remain at most eight;
child-verification workers remain two.
- More than eight members require the exact same canonical physical root
and complete source scope. A different root, new writer, changed source,
recovery owner or journal still refuses through the existing owners.
- Shared verification is limited to the held invocation fence. The
separate fresh final pass remains required after all captures join.
Different child sets and restart attempts do not share a local-byte
proof.
- The old saved membership and archive identities remain authoritative.
Older tools fail closed on the larger bound. No schema, archive or
provider format changes are included.

## Model Used

OpenAI Codex, GPT-6 family, with repository tools and code execution.
The runtime does not expose a more specific deployment identifier or
context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` /
`Closes: #` / `Refs: #` OR (b) described the issue in-PR following the
relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-09 20:03:25 -06:00
DottaandPaperclip 18697b8d21 perf(workspaces): share cohort child-source verification (#20418)
## Thinking Path

> - Paperclip must preserve agent workspace bytes during online
migration.
> - A cohort can contain several aliases of the same physical source.
> - The importer verifies imported children before capture and again
before publication.
> - The immutable-object cache avoids repeated downloads but still
recaptures local child bytes for every alias.
> - This pull request shares the initial proof within the held cohort
fence and checks fresh bytes once more after every capture joins.
> - The benefit is less repeated local archive work while child
mutations still block the whole cohort.
> - Required browser CI twice reproduced an independent stale
composer-mode bug. A focused regression proves that refreshed persisted
task mode was ignored after mount. The state repair follows task data
when there is no local override. Explicit local mode choices retain
their own generation so a completed send cannot clear the next draft’s
choice.

## Linked Issues or Issue Description

Refs: #20311, #20387, #20401, #20404.

**Current behavior**

Two aliases recapture the same imported child's local bytes four times,
even with one immutable object read. A larger alias cohort repeats this
multi-GB work for each member.

**Proposed behavior**

Share the successful initial verification for the same physical root,
Git roots and exact recorded child set or discovery mode. After all
captures join, run a separate fresh final verification for each root and
recorded child set. Keep every member's database and publication checks.

**Reason and benefit**

The local PostgreSQL fixture proves that two aliases now perform two
child recaptures instead of four. Hosted timing improvement has not been
measured.

## What Changed

- Add invocation-local initial and final proof maps to the existing
cohort child verifier.
- Retain the existing canonical source-owner check before and after
shared results.
- Defer cohort final nested checks until all captures finish, and join
every final reader before publication or fence release.
- Keep singleton verification, saved journal schemas, snapshot IDs,
archive bytes and per-member publication checks unchanged.
- Extend the existing real PostgreSQL alias fixture with a child
mutation during the last parent upload.
- Repair the required CI planning-mode race with a nullable local
override: absent override follows task data; a user selection belongs to
the draft. A successful send or confirmed resend/receipt clears it only
if no newer choice exists. Failed saves preserve the mode and restored
message. Queued-edit backup retains whether the mode was explicit. Leave
the browser assertion and timeouts unchanged.

## Verification

- Canonical Node24 worktree preflight and plugin SDK prerequisites
passed. Server TypeScript passed.
- Genuine unchanged-source PostgreSQL RED: two aliases performed four
local child recaptures instead of two (one failed, 22 intentionally not
selected). GREEN: five source-sharing, last-upload child mutation,
capture/cancellation join and actual outer-commit rollback controls
passed (18 intentionally not selected; 66.53s). The actual fence-loss
control also passed (one passed, 22 intentionally not selected; 38.13s).
Both wrappers completed cleanup.
- Migration source and tree bytes remain identical to locally qualified
`466837cfe997924c3794fd5d0e96134cab1cd04c`; later commits only repair
the composer and its tests.
- Required planning browser CI originally failed, then failed its one
unchanged-head retry, because the planning chip remained absent after
opening-question dismissal. Those failures remain recorded
([original](https://github.com/paperclipai/paperclip/actions/runs/42723622165/job/129523785985),
[retry](https://github.com/paperclipai/paperclip/actions/runs/42723622165/job/129534415590)).
The final-head identical case passes in 30.5s, with all 34 shard cases
passing ([final
CI](https://github.com/paperclipai/paperclip/actions/runs/42731309796/job/129550549775)).
Assertions and timeouts were not relaxed.
- Composer regressions reproduced stale loaded mode, optimistic-save
rollback, overwritten next-draft choices, and stale mode after confirmed
resend/receipt. All 145 composer tests now pass (2.88s). Canonical UI
TypeScript and token gates pass. The original planning browser case also
passed locally (15.5s; 26.9s including setup).
- The first final-head serialized-suite attempt was automatically
cancelled at its unchanged 20-minute job limit. Grouped Turbo output did
not expose an individual failing assertion; cause remains unknown
([retained
attempt](https://github.com/paperclipai/paperclip/actions/runs/42731309796/job/129550550373)).
One same-head diagnostic retry was a cache miss and executed all 419
heartbeat recovery tests successfully in 376.94s on the identical CI
checkout
([retry](https://github.com/paperclipai/paperclip/actions/runs/42731309796/job/129561196792)).
It is not classified as infrastructure or a known flake.
- Final exact head `ba1f85f18085dd0e84c594ed0cb68d5bf817c93c`: all 61
checks terminal (56 success, four skipped, one neutral); required CI is
green
([workflow](https://github.com/paperclipai/paperclip/actions/runs/42731309796)).
Fresh Greptile is 5/5 with every review thread resolved
([review](https://github.com/paperclipai/paperclip/pull/20418#issuecomment-6754705323)).

Focused commands:

```sh
node --import ./cli/node_modules/tsx/dist/loader.mjs packages/db/src/test-postgres-command.ts pnpm exec vitest run server/src/services/remote-workspace-cohort-migration.test.ts -t "shares the merged child verifier|joins active capture peers|keeps every authority and activity unpublished"
pnpm exec vitest run ui/src/components/task-chat/TaskChatComposer.test.tsx
pnpm --filter @paperclipai/server exec tsc --noEmit
pnpm --filter @paperclipai/ui typecheck
pnpm check:token-gates
```

## Risks

- Reuse is limited to one invocation under the same complete source and
finalizer fences. Different physical roots, Git roots, recorded child
sets and discovery mode have different keys. No local-byte proof is
stored durably or reused after restart.
- Initial and final maps are separate. A child change during any parent
upload must fail the fresh final verification before the atomic
publication transaction starts.
- Final readers all finish before an error can release the source fence.
Per-member database child rows, source fingerprints, journals, keys and
publication locks retain their existing checks.
- This reduces repeated verification for aliases. It does not change
capture concurrency, upload policy or immutable checkpoint validation.
Hosted speedup remains unverified.
- The CI repair must distinguish task data from explicit draft intent.
Only a successful send with the same choice generation clears its
override; a later draft selection and failed-save rollback retain it.
Regression tests cover these transitions. It adds no styling, request,
or persistence changes.

## Model Used

OpenAI GPT-6 (Codex), with tool use and code execution. The runtime does
not expose a more specific model identifier or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` /
`Closes: #` / `Refs: #` OR (b) described the issue in-PR following the
relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-09 19:28:31 -06:00
DottaandPaperclip bf818280f2 perf(workspaces): reserve cohort members concurrently (#20401)
## Thinking Path

> - Paperclip controls agent work and preserves its workspaces.
> - The online importer moves up to eight shared-source leaves under one
source fence.
> - Each member must have a durable reservation before capture starts.
> - Serial reservations repeat the same validation cost before any
archive work.
> - This pull request reserves members in parallel through their
existing independent transactions.
> - Every reservation finishes before an error can release the fence or
start capture.

## Linked Issues or Issue Description

Refs: #19873, #20387, #20394.

**Current behavior**

A measured six-member cohort spent about 110 seconds in serial
reservations. Each reservation keeps its existing source, task, recovery
and journal checks.

**Proposed behavior**

Start the bounded member reservations together. Join all outcomes,
retain member order, and refuse capture if any member fails. Keep the
exact partial-journal resume contract.

**Reason and benefit**

This removes serial metadata work before capture. Hosted speedup has not
been measured for this change.

## What Changed

- Replace only the cohort reservation caller loop with
`Promise.allSettled`.
- Preserve the existing per-member transaction, preassigned operation
ID, source fingerprint and all source fences.
- Extend the real PostgreSQL failure fixture to require overlapping
reservations and complete peer closure before fence release.
- Document the unchanged partial-journal and atomic publication
contract.

## Verification

- Canonical cached worktree preflight and plugin SDK prerequisites
passed on Node 24.22.2.
- The unchanged production baseline failed the new reservation overlap
assertion: one reader entered instead of two. The canonical PostgreSQL
wrapper cleaned up. The other 21 tests were intentionally not selected.
- The first baseline command failed before PostgreSQL because root-level
`tsx` resolution was absent. The corrected command uses the checkout's
explicit `cli/node_modules/tsx/dist/loader.mjs`.
- The four focused PostgreSQL controls passed before the final rebase:
overlap/join failure, exact partial-journal resume, parallel capture
with both cold restores, and atomic outer-commit rollback. Eighteen
tests were intentionally not selected. Server TypeScript passed.
- The final branch is based on merged #20394 at
`302ada90dfda0e8fbcd8a514d506d310c9843bcb`. The two directly affected
PostgreSQL reservation/resume controls passed again on final head
`06d436e45705da3b616a2b91725960f6e0ea52c0` (2 passed, 20 intentionally
not selected; 100.56 seconds). Server TypeScript passed again. The
canonical wrapper completed cleanup.

- The separate Ponytail review remains 4/5. Its nonblocking fan-out
warning was resolved as the documented phase contract; no source change
was required. Optional multi-error aggregation is outside this change.
- Fresh Greptile review is 5/5 on
`06d436e45705da3b616a2b91725960f6e0ea52c0`, with no actionable findings
or unresolved threads:
https://github.com/paperclipai/paperclip/pull/20401#issuecomment-6754025480.

- Exact-head CI is complete: 56 success, 4 skipped and 1 neutral, with
no failed or pending checks. Aggregate `ci / verify` passed:
https://github.com/paperclipai/paperclip/actions/runs/42718439355.
- Reproduce the final focused cases on Node 24: `node --import
./cli/node_modules/tsx/dist/loader.mjs
packages/db/src/test-postgres-command.ts pnpm exec vitest run
server/src/services/remote-workspace-cohort-migration.test.ts -t "joins
concurrent reservation validation readers|resumes the exact membership
after reservation stops halfway"`. Run `pnpm --filter
@paperclipai/server exec tsc --noEmit` for TypeScript.

## Risks

- API phase contract: `concurrency` limits capture workers. Preflight,
reservation and post-finalizer validation fan out over the exact
selected member set, with the canonical maximum of eight. The original
DB pool bounds connections. `concurrency: 1` does not promise one
metadata query or transaction at a time.
- Up to eight reservations can use independent pool connections at once.
Members that pass can leave durable journals when a peer fails. The
existing exact-member resume path retains those journals; this change
does not retry automatically.
- Capture and publication remain blocked until every reservation
settles. Publication stays serial in one transaction. Saved journal
schemas, singleton behavior and recovery proof checks do not change.
- This does not change concurrency inside the publication transaction.
One PostgreSQL connection serializes that SQL, and changing its row-lock
order has no measured benefit here.

## Model Used

OpenAI GPT-6 (Codex), with tool use and code execution. The runtime does
not expose a more specific model identifier or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` /
`Closes: #` / `Refs: #` OR (b) described the issue in-PR following the
relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-09 17:28:23 -06:00
DottaandPaperclip 9c634e42e4 fix(workspace): bound queued source-lock admission for imports (#20370)
## Thinking Path

> - Paperclip manages AI agents and their work.
> - Existing local workspaces need durable imports before a volume-free
deployment.
> - Imports must acquire the same source locks as workspace writers.
> - Try-only admission can repeatedly lose to background maintenance,
even when each writer finishes.
> - This pull request adds an optional bounded wait to the canonical
migration APIs.
> - Operators can join the source-lock queue while all existing
preservation checks remain active.

## Linked Issues or Issue Description

**What happened?**

An online import can repeatedly return `workspace_source_busy` while
background workspace maintenance reacquires its project source lock. The
import does not join the PostgreSQL lock queue.

**Expected behavior**

An administrative import can wait for a finite writer to finish. One
wait budget covers the complete source scope. Timeout or cancellation
must release acquired locks before the import reserves a journal.

**Steps to reproduce**

1. Hold the source lock for an eligible local workspace.
2. Start an online import while the writer holds that lock.
3. Release the writer. Before this change, the import has already
refused admission. With `sourceLockWaitMs`, the import acquires the
source scope and commits normally.

**Paperclip version or commit**

Reproduced on master `0e3fdebea8257bdff11573081c76b306f12db230`.

**Deployment mode**

External PostgreSQL with an online workspace import. Related
preservation work: https://github.com/paperclipai/paperclip/pull/20311.

## What Changed

- Add optional `sourceLockWaitMs` to singleton and cohort migration
APIs. Omitted options keep current admission behavior.
- Queue the sorted source keys with one decreasing wait budget. Abort
cancels only the dedicated backend identified by PID and backend start
time.
- Use an independent source transaction for queued singleton imports.
Keep lifecycle checks, fresh source checks, journal ownership, and
publication after commit.
- Document the canonical API option. Add real PostgreSQL importer
regressions for release, timeout, abort, and partial-key cleanup.
- Correct the inherited task-case lookup summary so the existing
search-discovery regression passes without changing its assertion or
search algorithm.

## Verification

- Baseline RED: the new singleton regression fails on the unchanged
product because the import returns busy without queueing.
- `pnpm exec vitest run
server/src/services/remote-workspace-migration.test.ts -t 'queued|bounds
the whole source queue'`: 20 passed, including five new cases.
- `pnpm exec vitest run
server/src/services/isolated-remote-workspace-source-lock.test.ts`: 4
passed.
- Server layer typecheck, `node scripts/check-module-boundaries.mjs`,
and `git diff --check` passed.
- Tests used Node 24.22.2 and the existing embedded PostgreSQL fixture.
Hosted rollout verification remains pending.
- CI build, typecheck, browser suites, all serialized suites, and the
other general shards passed. The final server shard failed only the
inherited `case-history-api-discovery.test.ts:14` search-ranking
assertion. Its unchanged local regression reproduces on the base and
passes 3/3 with a separate one-line route-summary correction. The shared
one-line summary correction is included in follow-up commit
`8163ffa9d4728134676b84c8c242a5bf62c802b5`; the unchanged case-discovery
suite passes 3/3 locally. The original CI attempt remains failed:
https://github.com/paperclipai/paperclip/actions/runs/42704666536/job/129458605783.

## Risks

Queued admission can hold earlier source keys while it waits for a later
key. The caller controls the finite budget. The abort path opens a
short-lived dedicated cancellation connection. If cancellation cannot
connect, PostgreSQL's remaining lock timeout still bounds the wait. This
option must be called at the root of administrative work; an external
scope that already holds the same source locks can cause
self-contention. The existing pre-held journal-resume path must keep its
omitted-option behavior.

## Model Used

OpenAI Codex, based on GPT-6, with tool use and code execution. The
exact deployment identifier and context-window size are not exposed in
this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` /
`Closes: #` / `Refs: #` OR (b) described the issue in-PR following the
relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-09 16:29:53 -06:00
DottaandClaude Opus 5.5 7f55323abb fix(recovery): end a remote workspace epoch whose writer died during preparation (#20346)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A task with a remote workspace prepares that workspace before the
run selects its runtime, and preparation marks the workspace epoch dirty
with the run as its writer
> - If the server is killed during preparation, the run never selected a
runtime and never started provider work, but the dirty epoch remains
> - Failed-workspace preservation, the path that resolves a dead
writer's dirty epoch, applies to native runs only, so nothing owned this
epoch
> - The dirty epoch kept the dead run's sandbox lease as a pending
export and held the workspace against the task's retry, so the task
never recovered
> - This pull request lets the ownership reconciler end such an epoch at
the last committed save
> - The benefit is that a restart during workspace preparation no longer
wedges the task

## Linked Issues or Issue Description

**What happened?**
In a restart-chaos wave on Daytona (a remote workspace that clones a
real repository), the controller was killed while each run's first
workspace save was in flight. That save is now settled by #20326. Each
run ended `process_lost` with a bounded retry, and both runs were still
in `execution_stage: preparing` with `runtime_mode_resolved_at: null`.
The workspace authority stayed `dirty: true`, `snapshotId: null`, with
the dead run as writer. The dead run's environment lease was proven
owner-gone but was refused by its settle as `workspace_export_pending`,
and the reconciler reported `I2_settled` violations. The retries stayed
`queued` until the test harness deleted the environment 21 minutes
later. Then they failed `remote_workspace_execution_required`, because
the environment was gone.

**Expected behavior**
When a writer dies before runtime selection, its unsaved files are only
the restored source and preparation output, because no agent turn ran.
Recovery must end that epoch at the last committed save, so the lease
settles and the retry restores the workspace into a new directory. A
writer that reached runtime selection must keep its existing recovery.

**Steps to reproduce**
1. Run a task whose agent uses Paperclip Runner with a remote sandbox
workspace.
2. Kill the server while the run is still preparing the workspace (for
example during its first save).
3. After the restart, the run ends `process_lost` and its retry is
queued, but the workspace stays dirty and owned by the dead run, the
lease stays active, and the retry never starts.

**Paperclip version or commit**
`master` at 251c6d5c69 (after #20326).

Refs #20326 (settles the save that was in flight when the controller
died).

## What Changed

- New `settlePreparationOnlyRemoteWorkspaceEpoch`
(`recovery/remote-workspace-unsaved-files-settlement.ts`), wired as the
settle of the existing `remote_workspace_unsaved_files` ownership kind.
Once the writer run is proven gone, it ends the epoch with the existing
`abandonUnverifiedRemoteWorkspaceEpoch` only when all of these are true:
  - the run is terminal;
  - its runtime was never resolved (`runtime_mode_resolved_at` is null);
  - it ended in `execution_stage: preparing`;
  - it has no native finalization;
- durable records show no work began: no `adapter.invoke` or
`adapter.progress` events, no `turn.*`, `provider.*` or
`native.process_*` events, and no recorded process id, process group,
native session or runner instance on the run;
  - the workspace authority is still dirty, with this run as writer;
  - no save for that epoch is still `running` or `pending`.
The epoch then advances with no writer and no location, and the
abandoned epoch is recorded in `remoteWorkspaceAbandonedEpochs` with an
activity entry.
- After that, the lease no longer matches a prepared source, so the
existing environment lease settle releases it and the workspace writer
hold is free for the retry.
- The ownership inventory row for this kind names the new settle.

## Verification

- Red before the fix, green after: `ends a dirty remote epoch whose
writer died during preparation, and keeps every other writer's files`
(`ownership-reconciliation-report.test.ts`). On master it settles 0
epochs. With the fix it ends only the preparation-only epoch, and keeps
the dirty epoch of a writer whose runtime was resolved, a writer that
reached execution, a live writer, and a writer with a save still in
flight.
- Red before the fix, green after, end to end: `lets a retry prepare the
workspace again after its first writer was killed during preparation`
(`remote-workspace-save-concurrency.test.ts`). It runs the real reserve
and prepare. The first writer's initial save is cut off, which leaves a
first epoch with no committed save (`snapshotId: null`). The run is
reaped as `process_lost` while still preparing, with its lease left
active. Two reconciliation passes then end the epoch and release the
lease. The retry reserves the workspace and prepares it again from the
project base in a new directory, then commits its own initial save. On
master, the epoch stays dirty.
- The safety cases also keep a preparation-stage writer with an
`adapter.invoke` event, and one with a recorded process id.
- Red before the fix, green after: `releases the lease and frees the
workspace for the retry once a preparation-only epoch ends`. Before the
passes, the lease counts as a prepared source and the workspace writer
hold is `held` for a retry. After two reconciliation passes, the lease
is no longer active, the prepared source is gone, and the writer hold is
`free`. On master the prepared source remains.
- The reconciler, retained-source fallback, remote workspace lifecycle,
save-concurrency, warm-handoff recovery, and retained orphan sandbox
suites pass (139 tests, 1 environment-gated file skipped). After
rebasing onto #20326, the reconciler, save-concurrency, retained-source
fallback, and controller instance lease suites pass (91 tests).
- `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` in `server` passed.
- Evidence: the runs in the wave's kept database all had
`runtime_mode_resolved_at: null` and `execution_stage: preparing`. Their
`runtime_mode: legacy` is only the column default, not a resolved
runtime.

## Risks

- This ends unsaved files without saving them. That is safe only because
no provider turn can have started before runtime selection, and the
guard requires the writer never to have resolved a runtime. A writer
that did resolve one keeps its existing preservation path.
- The abandoned root stays in the sandbox and is not reused. The next
preparation restores the last committed save, or the source, into a new
directory.
- No schema or API change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, local database inspection).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 14:12:51 -06:00
DottaandPaperclip c67909cb0f fix: keep tool approvals valid through OAuth refresh and delayed review (#20340)
## Thinking Path

> - Paperclip lets people manage agent work and approve app actions.
> - An approval must bind the action that the person reviewed.
> - OAuth access tokens can expire while that person is away.
> - The old target snapshot treated each token refresh as a new
authorization. It also imposed a one-hour review deadline.
> - This pull request separates token renewal from authorization
identity and keeps the human decision durable.
> - Current access, the reviewed arguments, meaningful target changes,
task closure, and execution receipts still control dispatch.
> - People can approve later without repeating a decision after routine
connection maintenance.

## Linked Issues or Issue Description

**What happened?**

A pending app action returned `approved_tool_target_changed` after its
OAuth tokens refreshed. The tool and connection settings were unchanged.
The same card also expired after one hour. A sign-in problem could
discard an accepted decision before provider dispatch.

**Expected behavior**

The original card remains reviewable after routine OAuth refresh and a
server restart. A later approval runs the exact reviewed action once. A
connection repair preserves the decision while Paperclip verifies the
same authorization. Meaningful changes require a new review.

**Steps to reproduce**

1. Connect an OAuth MCP app and configure an action to require approval.
2. Request the action, then refresh its access and refresh tokens before
approving.
3. Approve the original card. Before this change, it returns HTTP 409
even when the authorization is unchanged.
4. Wait more than one hour. Before this change, the card expires without
a task-specific deadline.

**Paperclip version or commit**

Reproduced from master at `76254ead2229`. The final branch is rebased
onto current master.

## What Changed

- Add versioned approval target comparison. New snapshots bind
credential identity epochs, consent, permissions, ownership, connection
configuration, tool contracts, and credential references. Authenticated
OAuth refresh preserves identity. Manual credential replacement does
not.
- Remove the implicit one-hour deadline from ordinary app action
reviews. Keep explicit deadlines and terminal-task expiry. Synchronize
the nullable interaction deadline in shared contracts and UI.
- Keep accepted, undispatched actions when sign-in is required. Show a
saved-approval repair state. Resume on committed connection changes
after target and access revalidation, without an idle retry loop.
- Preserve the existing execution claim, result receipt, and
uncertain-outcome no-replay rules.
- Explain meaningful target changes without inviting the person to retry
the stale card.
- Add a production task-thread Storybook example and document
compatibility and recovery behavior.

## Verification

- Red: production-refresher invalid-grant handling and formal-approval
withdrawal during a sign-in wait also reproduced as failing regressions
before their fixes.
- Red: before implementation, the local OAuth HTTP fixture failed for
both public and confidential clients with `approved_tool_target_changed`
after token refresh.
- Green: the extended fixture passes for public and confidential clients
with personal and shared grants. It covers actual local HTTP token
exchanges, multiple refreshes, a rebuilt gateway, an ended originating
run, approval after 25 hours, duplicate approval, preserved sharing
arguments, sign-in waiting, and event-driven repair. The gateways use
the production OAuth refresher; an invalid refresh preserves the
accepted decision before provider dispatch. Omitted-scope responses
cover equivalent scope-reporting changes. The provider is an isolated
local fixture, not a deployed Notebook service.
- Negative controls refuse changed consent, scopes, tool definition,
destination, and manual credential replacement before provider
execution.
- Existing gateway service, connected-tool acceptance, secret owner,
secret service, unsigned-review queue, and connection-health suites
passed. Explicit-deadline fixtures still prove expiry behavior,
including deadlines on accepted sign-in waits. A withdrawn formal
approval cancels a waiting action before connection recovery.
- Approval card, task-chat, error-copy and target-comparison tests
passed. Server and UI typechecks passed. Token gates and Storybook
catalog checks passed.
- Inspected the new saved-approval repair card in local Storybook using
the production task thread. It shows a reconnect link and no second
approval button or false running state.
- `pnpm verify --explain` returned exit 2 and requested broad CI
coverage. [Full
CI](https://github.com/paperclipai/paperclip/actions/runs/42677855507)
passed on `eb4f1c7caa9ee6e802c314ffc5eadec475a5de0c`, including build,
typecheck/release registry, general and serialized tests, and all eight
browser shards.
- Greptile reviewed the same final head at 5/5. Both findings from the
first review are fixed and their threads resolved; no open follow-ups
remain.

## Risks

- No database migration is needed. The new comparison reuses the vault
identity epoch. Only authenticated OAuth refresh code can preserve that
epoch; HTTP credential-write schemas cannot select that behavior.
- Earlier signed requests keep their original exact-version checks and
saved deadlines. Already-expired cards are not revived. A historical
card whose credentials already changed needs a fresh review.
- A new consent, account, permission set, destination, or manually
replaced credential must not inherit the old authorization. Reconnection
continues automatically only when revalidation proves the same saved
authorization.
- The interaction enrichment now permits a null deadline. Existing
string deadlines remain valid. The web client and shared validator
change together.
- This PR does not deploy to staging or perform a live Notebook
publication.

## Model Used

OpenAI Codex, GPT-6 family, with repository tools, code execution, and
browser inspection. The runtime does not expose the exact model ID,
context-window limit, or reasoning-mode label; those values are not
asserted.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-09 13:46:28 -06:00
DottaandClaude Opus 5.5 251c6d5c69 fix(recovery): settle a remote workspace save whose process died mid-save (#20326)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs in remote sandboxes save the workspace as snapshot
operations (`workspace_operations`, phase `remote_workspace_snapshot`).
Each save invocation is listed in `saveWork.activeIds` while it runs
> - When a controller process is killed during a save, its invocation
stays listed and the operation stays `running`
> - Nothing recorded which process ran the invocation, so the shared
ownership reconciler could not prove the owner gone and never settled
the save
> - The task's bounded retry then waited behind that save forever
> - This pull request records the owning process for each save
invocation, and lets the reconciler settle a save whose owner is proven
gone
> - The benefit is that a restart during a save no longer wedges the
task

## Linked Issues or Issue Description

**What happened?**
In a restart-chaos wave on Daytona with a real cloned repository, two
tasks each started their first remote snapshot save (`boundary:
initial`, epoch 1). A SIGKILL hit the controller during the save, before
any upload. Each run ended `process_lost` and got a bounded retry. The
retries stayed `queued` forever, the snapshot operations stayed
`running` with `saveWork.activeIds: [<dead invocation>]` and
`quiescedAt: null`, and the sandbox leases stayed active. The ownership
reconciler reported the saves as owner-gone but had no way to settle
them.

**Expected behavior**
After a restart, recovery must settle a remote snapshot save whose
owning process is gone. It is marked failed with provenance, so the
task's retry can continue. A save that is still running must stay
untouched.

**Steps to reproduce**
1. Run a native task with a remote workspace (for example Daytona with a
`git_repo` project source).
2. Kill the controller (SIGKILL) while the first workspace save is in
flight, before its upload.
3. After the restart, the run ends `process_lost` and its retry is
promoted to `queued`, but the save operation stays `running` and the
retry never starts.

**Paperclip version or commit**
`master` at 1b7fe33432.

Refs #20291 (the quiesced case: a save with no active invocation) and
#20233 (the same ownership pattern for local workspace commands).

## What Changed

- `saveRemoteWorkspace` records the controller process stamp of each
invocation in `saveWork.owners` when it registers the invocation. It
removes that entry when the invocation settles. A module-level set marks
invocations that are still running in this process
(`isRemoteWorkspaceSaveInvocationLive`).
- The ownership registry owns a `remote_snapshot_save` with active
invocations by that stamp. A save whose active invocations have no
stamps was written before this change and is owned by its run, as with
finalize attempts. A mix of stamps, or more than one process, is
unprovable.
- `settleAbandonedRemoteSnapshotSave` settles a save with active
invocations once its owner is proven gone:
- Process-owned: the proven stamp must still match every active
invocation, and none may be live in this process.
  - Legacy run-owned: the run must be terminal.
- Either way it clears the active invocations, records them as
`abandonedInvocationIds`, and marks the save `failed` with `lastFailure:
remote_workspace_save_abandoned`. The #20291 re-arm of blocked fallbacks
still applies.
- The controller instance lease counts running saves stamped with this
boot as held work, so a long save keeps its owner provably alive.
- The ownership inventory row for this kind describes the stamp.

## Verification

- Red before the fix, green after: `settles a remote snapshot save whose
process died mid-save, and keeps live, foreign or unprovable ones`
(`ownership-reconciliation-report.test.ts`). On master it settles 0
rows. With the fix it settles exactly two:
  - a save stamped by a dead process (proof `pid_dead`);
- the wave's shape: an unstamped active invocation on a `process_lost`
run.
It keeps five kinds of save running: a save stamped by this live
process, a stamp from another PID space, a legacy save on a live run, a
save with mixed stamps, and (in the older #20291 test) a stamped active
invocation.
- `stamps each save invocation with the process that runs it and clears
the stamp when it settles` (`remote-workspace-save-concurrency.test.ts`)
pauses a real save mid-flight. It checks the owner stamp and the
in-process live mark, then checks that both are cleared after the save
finishes.
- The reconciler, save-concurrency, remote workspace lifecycle, snapshot
retention, retained-source fallback, preparation gates, and controller
instance lease suites pass (153 tests, 1 environment-gated file
skipped).
- `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` in `server` passed.
- Not yet verified: the full Daytona wave replay that shows the queued
retry starts after the save settles. I asked the chaos harness owner to
replay it on this build.

## Risks

- A legacy save (written before this change) with an active invocation
is settled when its run is terminal. The finalize-attempt kind uses the
same rule. During a rolling deploy, an old-release process could still
be running such a save for a terminal run. If that save later succeeds,
its publication still marks the operation `succeeded`.
- If the retry in the wave is also held by the dead writer's dirty
workspace authority, settling the save may not be enough to start it.
The replay will show whether a second change is needed.
- No schema or API change. `saveWork.owners` is new optional metadata.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 13:27:44 -06:00
Dotta 1b7fe33432 fix(workspaces): preserve import verification across process restarts (#20311)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Online import preserves old local files before the web server can
use a disposable filesystem.
> - A parent import must authenticate saved child archives before it can
omit their directories.
> - The operation journal preserves import identity, but completed child
checks existed only in process memory.
> - A container restart therefore repeated downloads, decryption and
restoration that had already succeeded.
> - This pull request saves a versioned verification receipt on each
immutable child snapshot.
> - A replacement process can reuse those checks while it reacquires
locks and verifies current local files.

## Linked Issues or Issue Description

Refs #19854 for verification reuse within a live source fence. Refs
#19873 for cohort import. Refs #20298 for the separate remote-hold
admission fix. Searched related workspace verification and restart PRs;
none provides durable child verification.

**What happened?**

Replacing a server container interrupts a long online import. The
original operation and snapshot survive, but a later invocation
downloads, decrypts and restores completed child archives again. The
current cache ends with the source-lock owner.

**Expected behavior**

Completed immutable checks survive process and scratch loss. Resume
keeps the same operation and snapshot, reuses only correctly bound
verification, and publishes once. Current local files and Git state must
still be checked under fresh source locks.

**Steps to reproduce**

1. Import two nested child workspaces, then start the parent import.
2. Kill the importer after a child verification finishes. Remove its
scratch directory.
3. Resume in a new process. Before this fix, the completed child is
downloaded and restored again.

**Paperclip version or commit**

The regression fails on the previous migration verifier. The final
branch includes master at `189fefb99183953be37f0d1084275fbb0f35d6e2`.

**Deployment mode**

Managed Linux server with PostgreSQL and explicit S3 storage
configuration.

## What Changed

- Add optional opaque storage namespace identity. S3 binds it to the
explicit endpoint, region, bucket, normalized prefix and addressing
style. Implicit endpoints retain full verification.
- Save a versioned child-snapshot receipt only after canonical crypto,
manifest and file restoration succeed and a second HEAD confirms the
same strong ETag and size.
- Bind reuse to the complete snapshot reference, key material, child
lineage, source operation and storage namespace. Require a fresh strong
HEAD and canonical validation that the current secret provider can
unwrap the saved snapshot key.
- Keep current directory/Git capture and source-owner checks on every
invocation. Read-only verifier callers do not write checkpoints.
- Merge checkpoint metadata under a short row lock without changing
parent journal hashes, snapshot retention or update timestamps. Bound
reads and writes. Lock timeouts, statement timeouts, deadlocks and
serialization conflicts are safe cache misses. Other errors remain
fatal.
- Extend real PostgreSQL/Git/archive tests with process death, scratch
removal, resume, changed bindings and single publication.

## Verification

- Baseline RED: two import invocations read each child twice. With
durable receipts, each child is read once across both invocations.
- Real process regression passed on the final rebased head: SIGKILL the
owned importer after one checkpoint, delete all scratch, and resume in a
new Node process. It retains the same operation and snapshot, reads each
child once across both processes, and records one publication.
- 53 focused cases passed in 104.86 seconds on the final production
bytes before rebase. They cover existing nested/cohort behavior,
namespace identity, changed source/reference/key/lineage/ETag, unknown
proof versions, weak validators and bounded row contention.
- After clean rebase and stronger scratch/snapshot assertions, the real
process case passed again in 18.54 seconds. Server TypeScript, module
boundaries and diff checks passed on final head
`9da68ee7ce4a778201ef85e21a11c49eeea7bc16`. Required CI supplies
final-head broad coverage.
- The initial restricted-network dependency setup failed; unchanged
preflight passed with normal network access. Initial TypeScript failures
in test pipe assertions were corrected. No failed attempt is counted as
a pass.
- Review found a missing current-master-key check. A real restart
regression with all child checkpoints complete reproduced unsafe
publication after switching to a different valid master key. Follow-up
`7ea92c2abd9ffdb2e7dd81cecf62c8fc7a6e74a5` delegates to the canonical
snapshot-key owner before both durable and in-memory cache hits.
- The fixed process regressions passed 2/2. Ten key-authority and
existing identity/source guard cases passed. The wrong-key process now
refuses with zero additional child payload GETs, no publication, and
unchanged original journal/snapshot rows. Server typecheck, module
boundaries and diff checks passed on the follow-up.
- Review also found that transient database failures could make the
optional checkpoint cache stop a valid import. Read and save now fall
back only for SQLSTATE `55P03`, `57014`, `40P01`, and `40001`.
Connection loss, identity changes, crypto failures, programming errors
and cancellation remain fatal. Reads use a read-only transaction with
one-second lock and eight-second statement limits.
- The checkpoint database-fault regression failed two cases before the
fix. All four recoverable import cases then passed. The final selected
run passed 21 cases, including fatal-error and cancellation controls,
real row contention, both process-restart cases and key/identity guards.
Server typecheck and boundaries passed. Fault injection is limited to
the checkpoint database boundary; the importer, PostgreSQL, Git,
archives, storage and publication remain real.
- Latest master was merged without rewriting published history. Six real
import/process-restart cases passed again on final head
`8c8e4213a680ebb5c8d6864416bc3a4f271e1e76` in 38.03 seconds. [Required
CI](https://github.com/paperclipai/paperclip/actions/runs/42658248222)
finished with 54 successful checks and four intentional skips; Snyk
passed. [Fresh
review](https://github.com/paperclipai/paperclip/pull/20311#issuecomment-6745354381)
is 5/5 with all four threads addressed and resolved. Hosted verification
after normal release delivery remains pending.

## Risks

- A stale receipt could skip authentication of a different object.
Complete reference/lineage/namespace binding, fresh strong HEAD and
row-locked save validation prevent that reuse. Missing or unsupported
hints trigger full verification.
- Mutable local files can change between attempts. Their source locks
and fresh captures remain required; a checkpoint never certifies current
local bytes.
- Checkpoint persistence can contend with other snapshot owners. It has
one-second lock and eight-second statement limits; the four classified
transient SQL errors retain the normal fully verified path. Other
database errors stop the import so lost source ownership cannot be
hidden.
- No schema, archive format, public API or saved-run identity change.
Older snapshots without receipts remain readable and get full
verification. Existing data and recovery obligations stay intact.

## Model Used

OpenAI Codex, GPT-6.1 Sol (`gpt-6.1-sol`), with reasoning, repository
tools and code execution. Context-window size is not exposed by the
runtime.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-11-09 11:03:48 -06:00
DottaandClaude Opus 5.5 099adb2728 fix(recovery): settle a remote workspace save that stopped without finishing (#20291)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs in remote sandboxes save the workspace as snapshots.
Each save is a `workspace_operations` row with phase
`remote_workspace_snapshot`
> - The ownership registry lists that kind as in-progress work, but it
had no settle step
> - When a restart cut off a failing final save, the row stayed
`running` after its invocations had all stopped and its run had ended
> - The retained-source fallback refuses while any workspace operation
is running, so a task whose saved files could not be verified could
never continue from its last save
> - This pull request lets the ownership reconciler settle that row as
failed
> - The benefit is that the automatic fallback can continue such a task
without any user action

## Linked Issues or Issue Description

**What happened?**
A native run in a remote sandbox finished its turn. Its final workspace
save failed because the sandbox disk was full ("No space left on device"
in the snapshot copy). A graceful restart then interrupted the retry.
The run ended as `native_workspace_sync_out_retry_exhausted`. Its
`remote_workspace_snapshot` row kept `status: running` for six days,
with `saveWork: { activeIds: [], unknown: false, quiescedAt: <time> }`
and `exportCleanup: { state: "pending" }`. The retained-source fallback
checked the task and refused with `workspace_operation_running`, so the
task stayed blocked.

**Expected behavior**
A save with no active or unknown invocation, whose run is terminal, is
finished work. The reconciler must mark it failed, so the fallback can
continue from the last committed save. A save that might still be
running must stay untouched.

**Steps to reproduce**
1. Run a native task in a remote sandbox. Make its final save fail, then
restart the server during the retry, so the save's work is quiesced and
the operation stays `running`.
2. Let the task's recovery refuse with an unverifiable-source failure
(for example `allocation_ownership_ambiguous`).
3. The retained-source fallback refuses with
`workspace_operation_running` on every check, and the task never
continues.

**Paperclip version or commit**
`master` at f7c85bca18.

Refs #20282 (the other cause of the same stuck task: a false ownership
refusal).

## What Changed

- `settleAbandonedRemoteSnapshotSave`
(`recovery/workspace-finalize-attempt-settlement.ts`) settles a
`remote_workspace_snapshot` row in `running` or `pending` only when all
of these are true:
- its save work is quiesced (`activeIds` is empty and `unknown` is
false);
  - its run is terminal.
It sets `status: failed`, records `abandonedSave` provenance and
`lastFailure: remote_workspace_save_abandoned`, keeps `saveWork` and
`exportCleanup`, writes a `workspace_operation.reconciled` activity
entry, and wakes the workspace's terminal work.
- The ownership registry wires it as the settle of
`remote_snapshot_save`.
- In the same transaction it re-arms blocked fallbacks. For every open
hold whose fallback check last refused with
`workspace_operation_running` and whose source run used this save's
workspace, it clears the check and wakes the workspace-recovery queue. A
hold that had already spent its 12 checks (no next check) continues as
well.
- The settle uses the save path's own `readSaveWork`, now exported, so
it never settles save work that the save path would treat as invalid.
- Because the row now has a `lastFailure`, `retireQuiescedSaves` retires
it like any failed save when a later final save commits. A later retry
of the same save boundary reopens a failed row, as before.
- The ownership inventory row for this kind names the new settle.

## Verification

- Red before the fix, green after: `settles a remote snapshot save that
stopped without finishing after its run ended, and keeps open or live
ones` (`ownership-reconciliation-report.test.ts`). Before: `settled: 0`.
After: only the abandoned save settles. Saves with a live run, an active
invocation, or unknown save work stay `running`.
- Red before the fix, green after, end to end: `continues once the
reconciler settles a final save a restart left running after its run
ended` (`workspace-recovery-fallback.test.ts`). The fallback first
refuses with `workspace_operation_running`, and the test then spends its
check budget (12 checks, no next check), so the hold drops out of
discovery. After one ownership reconciliation pass, the save is `failed`
and the check is cleared. The worker then resolves the hold with
`retained_source_unverifiable_continued_from_last_saved`. Without the
re-arm, this test fails.
- The reconciler test also covers save work that is not quiesced
(`quiescedAt: null`), save work that is invalid (`quiescedAt: ""`), and
a `pending` save.
- `workspace-recovery-fallback.test.ts` (16 tests),
`ownership-reconciliation-report.test.ts`, and the remote workspace
lifecycle, save-concurrency, and snapshot-retention suites pass (80
tests, 1 environment-gated file skipped).
- `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` in `server` passed.

## Risks

- A save in another process registers its invocation in `activeIds` in
the same row-locked transaction before it runs. The settle locks the row
and requires `activeIds` to be empty and `unknown` to be false, so it
cannot fail a save that is still working.
- A save whose process died with its invocation still in `activeIds`
stays `running`. That case needs an owner-death proof, which is out of
scope here.
- No schema or API change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, read-only browser diagnostics).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 10:26:04 -06:00
Dotta 5c73633a8f fix(workspaces): preserve remote recovery holds during online import (#20298)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Online workspace import preserves old controller-local files while
unrelated remote work continues.
> - A task can have an old local checkout and a separate durable remote
workspace.
> - A terminal remote run can retain a file-recovery hold on that task.
> - The legacy source checker rejects remote references, so that hold
blocks independent local imports.
> - This pull request proves that the hold belongs only to the separate
remote source and fences its finalizer during import.
> - Old files can migrate while the remote recovery obligation stays
intact.

## Linked Issues or Issue Description

Refs #19873 and #19854 for the existing cohort and source-verification
paths. Searched related online migration and remote recovery PRs; no
duplicate fix was found.

**What happened?**

Online import rejects a legacy Git checkout with
`online_workspace_source_scope_unproven` when a same-task failed run has
a prepared reference to a separate durable remote workspace. The legacy
run reader rejects remote references before the hold can be classified.

**Expected behavior**

Independent local files can import under their source fences. The
separate remote workspace, dirty writer, allocation, accepted result and
recovery hold must stay intact.

**Steps to reproduce**

1. Create related local Git workspaces.
2. Record a terminal failed native run and an exhausted file-recovery
hold for a separate durable remote workspace on the same task.
3. Import the local leaf online. The old checker refuses before capture.

**Paperclip version or commit**

Reproduced on master at `052c638b519ebb15f81e4fcd1a96adf6cd3b2f10`.

**Deployment mode**

Managed server with PostgreSQL and remote Runner execution.

## What Changed

- Add a distinct remote-only preservation proof. Keep remote runs
excluded from legacy export sources.
- Use the canonical source inspector and prepared-reference matcher.
Bind the run, finalizer, action, task, accepted result, authority,
allocation and prior committed snapshot.
- Acquire the held run's finalizer fence through the existing sorted
lock path. Existing source-proof reinspection rejects changes before
publication.
- Preserve previous-release prepared references with an older baseline
snapshot. Do not require that snapshot's epoch or source run to equal
the current run.
- Add real PostgreSQL import, authenticated cold restore, preservation,
refusal, lock-contention and publication-race regressions. Document the
behavior.

## Verification

- Genuine baseline failure on unchanged product: the real importer
refuses with `online_workspace_source_scope_unproven`.
- Final focused regression file: 23 tests passed in 13.19 seconds
through the canonical isolated PostgreSQL wrapper on Node 24.22.2.
- 24 existing affected migration/recovery tests passed against the same
final production bytes. Another 83 tests were explicitly excluded by the
focused selection; no full-suite pass is claimed.
- The positive imports local bytes, confirms the remote finalizer fence
is held during upload, leaves all remote records unchanged, parks the
original repository and authenticates/restores the archive.
- Refusal and race cases cover malformed, foreign, same-local and mixed
references; active owners/writers; changed allocation, result, task,
action, authority and baseline; and finalizer contention.
- Server TypeScript check, module-boundary check and `git diff --check`
passed.
- An intermediate run had one incorrect assertion that expected an
explicit empty legacy-source array. The canonical journal omits that
empty field. The final assertion accepts absence and still forbids any
exported remote source. Final relevant results are 23 new plus 24
existing passes, rather than a claim that the earlier invocation passed
in full.
- Required CI passed on head `6a278e717db2b5f614b0a595f99bfd0fb5430227`:
55 successful checks and 4 intentional skips. [CI
run](https://github.com/paperclipai/paperclip/actions/runs/42650025033).
Greptile reviewed that head at 5/5 with no actionable findings or open
inline/review comments. Hosted continuation remains pending normal
delivery.

## Risks

- A false disjointness proof could admit an unsafe local import. Strict
company/task/source binding, the canonical inspector, finalizer fencing
and publication revalidation retain refusal on uncertainty.
- This does not recover or settle remote files. Dirty authority and the
original recovery obligation remain unchanged.
- No schema, public API, archive format or saved-run identity changes.
Existing persisted state carries forward.

## Model Used

OpenAI Codex, GPT-6.1 Sol (`gpt-6.1-sol`), with reasoning, repository
tools and code execution. The runtime does not expose the context-window
size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-11-09 10:13:41 -06:00
DottaandClaude Opus 5.5 ac0d0374f2 perf(issue-detail): measure task opens on real instances (#20276)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - People open a task from their issues list to see what an agent is
doing and to reply
> - On a busy staging instance, the composer and the latest messages
take 1.1–2.0 s to appear after a click. For one task with a slow run
log, they took 13.9 s
> - Local fixtures do not reproduce this. They do not have real network
round trips, real server contention, or real object-store latency
> - This pull request adds the measurement we need: a browser harness
that times a task open on a real instance, and a Server-Timing entry on
every API response that separates server time from network time
> - It also records the staging baseline and the improvement plan
> - The benefit is that each later performance change can be measured on
real data with the same tool, before and after deploy

## Linked Issues or Issue Description

**Problem or motivation**
Opening a task from the issues list is slow on instances with real data.
The composer and the latest chat messages can take seconds to appear. We
could not measure this on staging in a repeatable way. We also could not
tell server time from network and queueing time for most API requests.

**Proposed solution**
Add a reusable browser harness for task opens, and add a `paperclip_api`
Server-Timing entry to every API response.

**Alternatives considered**
Local Playwright fixtures only. They do not reproduce real network round
trips, server contention or object-store latency, so they did not show
the delays that people see. Per-endpoint Server-Timing only. It does not
cover the many other requests that run during a task open.

**Roadmap alignment**
This is measurement for existing task-loading behaviour. It does not add
product features. The staging baseline and the measured causes are in
`doc/plans/2026-11-09-task-loading-performance.md`. Follow-up pull
requests change the loading behaviour itself, starting with #20277.

## What Changed

- Added `server/src/middleware/api-server-timing.ts`. Every `/api`
response now has a `paperclip_api;dur=<ms>` Server-Timing entry. The
value is the server time from request arrival until headers are written,
including admission waits. The entry is appended, so the existing
`paperclip_issue` phases stay first.
- Registered the middleware first in `server/src/app.ts`.
- Added `tests/perf/issue-detail/staging-click-harness.js`. It clicks a
task row and records, frame by frame, when the composer and the latest
message become visible, layout shifts, movement of the latest message,
and the requests that run before the page is ready. The summary labels
each sample instead of using the task reference. It replaces query
values and every path segment that contains a digit. A fetch that is
still running when the sample ends is listed as pending.
- Documented the harness in `tests/perf/issue-detail/README.md`.
- Added the dated plan
`doc/plans/2026-11-09-task-loading-performance.md` with the staging
baseline, the causes, and the prioritized slices.

## Verification

- `cd server && npx vitest run src/__tests__/api-server-timing.test.ts`.
3 tests pass. They cover the JSON responses, the time spent in earlier
middleware, preserving route phases, and explicit `writeHead` calls.
- `tests/perf/issue-detail/issue-detail.perf.spec.ts` reads the first
`dur=` value. That value is still `paperclip_issue`.
- The harness was used on staging to capture the baseline in the plan
document.

## Risks

- Low risk. The header has only a duration, with no request, user or
company data.
- The middleware wraps `res.writeHead` once for each request. This is
the same pattern that `on-headers` uses.
- Server-Timing is visible to same-origin scripts only. No
`Timing-Allow-Origin` header is added.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code
with tool use (shell, browser automation, file editing).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 09:06:37 -06:00