mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 13:01:05 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - `AGENTS.md` tells every contributor and agent to read `doc/DEVELOPING.md` for setup, commands, testing, and worktrees > - That file grew to 5,495 lines. Most of it was notes on how one subsystem behaves, added one pull request at a time > - Readers could not find the commands. Agents also read behavior notes as rules > - This pull request cuts `doc/DEVELOPING.md` to a 660-line setup and workflow guide. It moves the subsystem notes to focused docs > - The benefit is a guide that a person or an agent can read in full, and a clear place for each kind of note ## Linked Issues or Issue Description **Issue type** Outdated or confusing documentation. **Where is the issue?** `doc/DEVELOPING.md` **What's wrong?** The file was 5,495 lines. It opened with 112 lines about case handoffs before it reached Prerequisites. Unrelated sections sat below the "OpenClaw Docker UI One-Command Script" heading. Statements about current behavior read like policy. **Suggested fix** Keep setup, commands, and workflow in `doc/DEVELOPING.md`. Move each subsystem note to a doc for that subsystem. Add an index. ## What Changed - `doc/DEVELOPING.md` is now 660 lines: prerequisites, start dev, health checks, local run, test drive, Docker, instance layout, database, storage, secrets, worktrees, test commands, Storybook, CLI, and an index of the other docs. - 15 new docs hold the moved text without wording changes: `ci-workflows`, `test-infrastructure`, `worktrees`, `hot-restart-deploys`, `heartbeat-internals`, `runner-internals`, `workspace-repositories`, `workspace-storage`, `skills-and-teams-catalog`, `ui-internals`, `case-handoffs`, `chat-connector-development`, `instance-configuration`, `integration-api-notes`, `database-backups`. - "Publish a branch Storybook" moved to `doc/STORYBOOK-DEPLOYMENT.md`. The privacy rollout notes moved to `doc/ISSUE-PRIVACY.md`. - Deleted about 280 lines: the story-by-story Storybook notes, "Agent Artifact Uploads" (a copy of `doc/AGENT-ARTIFACTS.md`), two pointer-only sections, "Workspace feature checks (2026-10-09)", and "Chat and task navigation caching". The test-drive build-lock paragraph is now four lines. - Repointed every inbound link and every script or workflow comment that named a moved section. - `AGENTS.md` now says where a behavior note goes, so the guide does not grow again. ## Verification - A script split the file by line range. It asserts that each of the 5,495 source lines is kept, moved, or dropped exactly one time. - A link check on every changed Markdown file found 0 missing files and 0 missing anchors. - `node scripts/check-maps.mjs` passes. - `node --test .github/scripts/tests/*.test.mjs`: 808 pass, 0 fail. - `node --check` passes on the three scripts with changed comments. - `pnpm verify` was not run. This worktree has no installed dependencies, and the change has no runtime code. ## Risks - Low risk. The change is documentation and comments only. - A link to a moved section from outside this repository will land on the top of `doc/DEVELOPING.md`. - The new docs are not in the audited prose list in `.github/scripts/ci-inputs.mjs`. An edit to one of them runs full CI. An edit to `doc/DEVELOPING.md` does not. This PR does not change that list. ## Model Used - Claude Fable 5.2 (`claude-fable-5-2`), Claude Code desktop, with tool use. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [ ] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
547 lines
28 KiB
Markdown
547 lines
28 KiB
Markdown
# Test infrastructure
|
|
|
|
Internals of the local and CI test tooling, plus the benchmarks and live smoke tests. For everyday commands, see [DEVELOPING.md](DEVELOPING.md#test-commands).
|
|
|
|
## Runner transport suite
|
|
|
|
The Runner transport suite keeps attachment recovery, warm attachment, live
|
|
adoption, and suspension in separate test files. Vitest can schedule these
|
|
independent fixtures on its existing file workers; cases within each file stay
|
|
serial. The package sequencer starts the ten files measured above 50 seconds in
|
|
the November 4, 2026 AWS run first, ordered by approximate duration. This includes
|
|
transport deadlines, settlement, provider launch, recovery, and streaming waits,
|
|
so fresh CI runners do not have to infer duration from file size. Package CI uses
|
|
up to four workers, bounded by available CPUs; local runs retain Vitest defaults.
|
|
Vitest's file selection and shard membership are unchanged. Real lease/deadline
|
|
waits and assertions are unchanged. Run the complete
|
|
family, including the separate deadline and settlement regressions, with:
|
|
|
|
```sh
|
|
pnpm --filter @paperclipai/paperclip-runner exec vitest run runnerd-codex-transport
|
|
```
|
|
|
|
## Embedded Postgres template cache
|
|
|
|
To run the suite with one owned PostgreSQL server and a separate disposable
|
|
database for each fixture, use:
|
|
|
|
```sh
|
|
node cli/node_modules/tsx/dist/cli.mjs packages/db/src/test-postgres-command.ts
|
|
node cli/node_modules/tsx/dist/cli.mjs packages/db/src/test-postgres-command.ts -- pnpm exec vitest run --project @paperclipai/db
|
|
```
|
|
|
|
The command starts an embedded server, prepares a migrated database template,
|
|
passes it to the child command, and waits for server shutdown before returning.
|
|
It preserves the child command's exit status. On cancellation it stops and waits
|
|
for the test process group before shutting down PostgreSQL. Database, server,
|
|
and CLI fixtures all use the common implementation. Tests of embedded startup itself continue to
|
|
start dedicated clusters.
|
|
|
|
To reuse a disposable server that you already manage, set
|
|
`PAPERCLIP_TEST_POSTGRES_ADMIN_URL` to its administrative PostgreSQL URL. The role
|
|
must be able to create databases. Each fixture creates a random database and
|
|
closes only its clients before dropping it. It never stops the external server.
|
|
`DATABASE_URL` does not select this test path. The optional
|
|
`PAPERCLIP_TEST_POSTGRES_TEMPLATE` can name an existing migrated database with
|
|
the form `paperclip_test_template_<hex>`; `0` disables template reuse.
|
|
|
|
On a supported platform, PostgreSQL startup failures fail test collection.
|
|
Resource exhaustion, unavailable configured servers, and broken native binaries
|
|
do not make integration tests skip. A genuinely unsupported platform or CPU can
|
|
still skip embedded PostgreSQL coverage. Startup retries apply only to port
|
|
collisions. Teardown waits for process exit, requests fast shutdown first, and
|
|
uses immediate PostgreSQL shutdown if needed. A failed stop fails the hook and
|
|
retains its data directory. Worker termination also requests owned server
|
|
shutdown. A forced `SIGKILL` can bypass these handlers; the separate host cleanup
|
|
job remains the backstop for stale macOS semaphore sets.
|
|
Database and CLI hooks use the same 30-second deadline as server hooks. This
|
|
allows the ten-second shutdown protocol and client/database cleanup to finish
|
|
before the test runner ends a worker. A shutdown timeout still fails the hook.
|
|
Template publication uses `pg_ctl` to request clean fast shutdown on every
|
|
supported platform and allows thirty seconds for it. A template is never
|
|
published after immediate shutdown or a crash. Cancelling the shared-server
|
|
command also waits for its owned cluster-directory cleanup after the test
|
|
processes and PostgreSQL exit. Direct worker termination stops PostgreSQL but
|
|
retains its disposable files for inspection; the synchronous exit guard cannot
|
|
await directory removal.
|
|
Stopped cluster directories are removed asynchronously, and cleanup awaits their
|
|
removal. Large suites can leave many relation files; deleting them must not block
|
|
the worker's event loop. The idle-sleep suite uses the existing 90-second fixture
|
|
budget for its final directory cleanup after hundreds of table truncations.
|
|
|
|
Suites that call `startEmbeddedPostgresTestDatabase` (from `@paperclipai/db`)
|
|
without an administrative URL get their own embedded Postgres cluster. The first such suite on a host builds
|
|
one fully migrated cluster and publishes its data directory under
|
|
`$TMPDIR/paperclip-embedded-postgres-templates-<uid>/<key>`. Every later suite
|
|
copies that template and starts Postgres on the copy instead of running
|
|
`initdb` and all migrations again, and the `getEmbeddedPostgresTestSupport`
|
|
probe reuses it instead of starting a throwaway cluster. The key hashes the
|
|
embedded-postgres version, the initdb flags, and every migration file, so a
|
|
changed migration builds a new template. Each suite still runs
|
|
`applyPendingMigrations` on its copy, which is a no-op when the template is
|
|
current. Templates for other keys are removed after seven days without use.
|
|
|
|
A template is a Postgres data directory, and starting it executes its
|
|
configuration, so the cache is private to the current OS user: the root is
|
|
created with mode `0700` and is only used when it is a real directory (not a
|
|
symlink) owned by the current user with no group or world write bit, and each
|
|
template is checked the same way before it is copied. A root that fails those
|
|
checks (for example one pre-created by another user in a shared `/tmp`) is left
|
|
alone and every suite in that process uses a fresh cluster.
|
|
|
|
Set `PAPERCLIP_TEST_POSTGRES_TEMPLATE=0` to bypass the cache and give every
|
|
suite a fresh `initdb`, for example when debugging cluster startup itself. A
|
|
failure on the template path (root checks, build, copy, start, or publish)
|
|
falls back to the fresh path automatically, so the cache can make a suite
|
|
slower but never fail it or skip it: a migration that cannot apply still fails
|
|
the suite on the fresh path.
|
|
|
|
## Typecheck memory
|
|
|
|
TypeScript 7 is a native binary, so `NODE_OPTIONS=--max-old-space-size` does not
|
|
limit `tsc`. Its peak memory depends on how much source one process checks and on
|
|
the number of checkers, because each checker keeps its own copy of the types.
|
|
|
|
`pnpm --filter @paperclipai/server typecheck` does not check one program.
|
|
`server/scripts/typecheck-layers.mjs` reads the import graph and runs one `tsc`
|
|
process per layer. Each layer reads the earlier layers as declarations:
|
|
|
|
| Layer | Files |
|
|
| --- | --- |
|
|
| `below` | Files the largest import cycle depends on, and unrelated files |
|
|
| `cycle` | The largest import cycle |
|
|
| `above` | Files that depend on the cycle |
|
|
| `tests` | Test files outside `src/__tests__` |
|
|
| `test-tree-0` to `test-tree-2` | `src/__tests__`, in shards |
|
|
|
|
The generated configs and declarations go in
|
|
`server/node_modules/.cache/paperclip-typecheck`. You do not maintain a layer list.
|
|
`server/tsconfig.json` remains the single program for editors and `pnpm build`.
|
|
The server typecheck builds only the Runner's TypeScript output. Use
|
|
`pnpm --filter @paperclipai/paperclip-runner build` when a local check needs the
|
|
Runner binary.
|
|
|
|
`node server/scripts/typecheck-layers.mjs --layer <name>` checks one layer
|
|
against the configs and declarations that an earlier `--declarations-only` run
|
|
left in that directory. It does not write them again. Stale declarations pass
|
|
silently, so use it only straight after `--declarations-only` on the same commit.
|
|
|
|
If a compiler child terminates on a signal, the wrapper reports the signal,
|
|
layer, command, and working directory on stderr, then stops with a nonzero exit.
|
|
Compiler stdout and stderr remain visible. A signal such as `SIGKILL` alone does
|
|
not establish that the process ran out of memory.
|
|
|
|
`src/__tests__/tsconfig.json` holds the options for the test-tree layers, so an
|
|
editor and the typecheck report the same errors. Server tests reach Daytona plugin
|
|
source, but the root install does not install that plugin's dependencies. So this
|
|
config maps `@daytonaio/sdk` to `server/scripts/typecheck-stubs/daytona-sdk.d.ts`.
|
|
The plugin's own project checks that source against the real package.
|
|
|
|
If the memory available to the task is limited, use this command:
|
|
|
|
```sh
|
|
node scripts/typecheck-low-memory.mjs
|
|
```
|
|
|
|
It peaks at about 3.8 GiB, against about 5.5 GiB for `pnpm -r typecheck` on 16
|
|
cores. It also takes about twice as long.
|
|
|
|
It checks one package at a time and uses one thread for the server layers and the
|
|
UI. Do not add `GOMEMLIMIT` to this path. It saves at most 300 MiB here, and it
|
|
slows `tsc` by up to 20 times when the live heap goes above the limit.
|
|
`pnpm build` still checks the whole server program and needs more than 8 GiB. The
|
|
Docker image sets `PAPERCLIP_TSC_EMIT_ONLY=1` for that build, which emits
|
|
JavaScript without checks or declarations.
|
|
|
|
Test projects outside `pnpm -r` import server source through relative paths. Each
|
|
project's `tsconfig.layers.json` reads that source as layer declarations. This
|
|
command builds the declarations and then checks every project:
|
|
|
|
```sh
|
|
node scripts/typecheck-test-projects.mjs
|
|
```
|
|
|
|
Name a project to check only that project. `pnpm test:e2e:runner:typecheck` and
|
|
`pnpm test:lifecycle-baseline:typecheck` do this for their projects. If the memory
|
|
available to the task is limited, add `--low-memory`, or set
|
|
`PAPERCLIP_TYPECHECK_LOW_MEMORY=1` for the pnpm commands.
|
|
|
|
The command fails if a project loads server source, because that is what makes
|
|
these programs large. It does not limit memory. CI runs it with
|
|
`--reuse-declarations` after the server declarations of the same run, so it does
|
|
not build the declarations again. Do not use that option when the declarations
|
|
can be stale.
|
|
|
|
Add each new test project that imports server source to `PROJECTS` in
|
|
`scripts/typecheck-test-projects.mjs`.
|
|
|
|
Keep the project references out of each project's `tsconfig.json`. Vitest reads
|
|
that file, and it fails when a referenced config does not exist yet. The
|
|
`tsconfig.json` files remain single programs for editors, and
|
|
`tests/runner-e2e/native-completion-checks.mjs` still checks that project as one
|
|
program. That program peaks at about 15 GiB at default settings and about 8 GiB on
|
|
one thread.
|
|
|
|
## Typecheck test files outside the build
|
|
|
|
Most package tsconfigs exclude tests, and Vitest strips types, so a type error in
|
|
one of those tests fails nothing. `scripts/typecheck-test-programs.mjs` reads them
|
|
through separate programs, usually a `tsconfig.test.json` beside the package's
|
|
`tsconfig.json`:
|
|
|
|
```sh
|
|
node scripts/typecheck-test-programs.mjs --low-memory
|
|
node scripts/typecheck-test-programs.mjs packages/paperclip-runner/tsconfig.test.json
|
|
```
|
|
|
|
The first form runs in the required `Typecheck + Release Registry` job, split in
|
|
two: `--prepare-only` installs the sandbox providers and builds what the programs
|
|
read, and `--skip-prepare` then only checks. It checks
|
|
one program at a time and the largest peaks below 2 GiB. It then fails if a tracked
|
|
`*.test.ts(x)` or `*.spec.ts(x)` file is in no program. The compiler's file list
|
|
decides that, because `include` silently drops `foo.test.tsx` when `foo.test.ts`
|
|
exists.
|
|
|
|
To add a test directory, add it to the `include` of the package's
|
|
`tsconfig.test.json`. To add a package, add a program to `PROGRAMS` in the script.
|
|
|
|
Sandbox providers are outside the root workspace, so the root install has none of
|
|
their SDKs. `scripts/typecheck-sandbox-providers` is a small workspace that lists
|
|
them and holds one lockfile. Its install replaces any standalone `node_modules` in
|
|
those providers. After you change a provider's `package.json`, do not refresh or
|
|
commit that lockfile. `typecheck:test-programs` resolves a stale lockfile in place
|
|
for the run, and the lockfile refresh workflow commits the new one after merge (see
|
|
[Dependency Lockfile Policy](ci-workflows.md#dependency-lockfile-policy)).
|
|
|
|
Do not add a lockfile inside a provider.
|
|
`scripts/build-standalone-public-packages.mjs` uses a frozen install when one
|
|
exists, which changes what a release resolves. Daytona already has one; the
|
|
release build resolves it in place when it is stale.
|
|
|
|
Daytona is the one exception: it commits its own lockfile, so the workspace
|
|
leaves it out and the script installs it from that lockfile (`LOCKED_PROVIDERS`).
|
|
The same rule applies to it: do not refresh or commit that lockfile. The script
|
|
resolves a stale one in place for the run, and the lockfile refresh workflow
|
|
commits the new one after merge.
|
|
|
|
Dependabot updates the providers through this workspace (see
|
|
`.github/dependabot.yml`), and Daytona through a second entry. Dependabot does
|
|
not expand a workspace glob that starts with `../`, so
|
|
`scripts/typecheck-sandbox-providers/pnpm-workspace.yaml` lists each provider by
|
|
its full path. When you add a provider, add it there and to `SANDBOX_PROVIDERS`
|
|
in `scripts/typecheck-test-programs.mjs`. A test checks that the two lists match.
|
|
|
|
## Sandbox provider tests
|
|
|
|
`pnpm test:run` does not reach the sandbox providers, because they are outside the
|
|
root workspace. `scripts/test-sandbox-providers.mjs` installs them the same way as
|
|
the typecheck above, then runs every provider's `pnpm test`, four at a time:
|
|
|
|
```sh
|
|
node scripts/test-sandbox-providers.mjs
|
|
```
|
|
|
|
The required `Sandbox provider tests` job runs it when a pull request changes a
|
|
provider, the plugin SDK, or `packages/shared`, which the SDK imports. Those paths
|
|
are the `//#ci:sandbox-providers` inputs in `turbo.json`. Global CI inputs, such as
|
|
`scripts/**` or a `package.json`, select every job. If the SDK gets a new workspace
|
|
dependency, add that package to the inputs. A test in
|
|
`.github/scripts/tests/ci-plan.test.mjs` fails until you do.
|
|
|
|
Some Daytona file-sync tests need GNU `tar` and coreutils. They skip on macOS
|
|
unless those tools are on `PATH`, so they run only on Linux. The job turns off
|
|
core dumps: a core file in an exported directory changes the transferred file count.
|
|
|
|
## Reused Runner build in the server build
|
|
|
|
The server build vendors `packages/paperclip-runner/dist`, so its
|
|
`prepare:runner-vendor` step has to make sure that directory is a current
|
|
build. It used to run the Runner's full `build` every time. Now the Runner's
|
|
`build` brackets itself with `scripts/record-build-inputs.mjs`: `--begin`, its
|
|
first step, forgets the previous stamp and records a digest of the build's
|
|
inputs (every file under `packages/paperclip-runner` except `dist`,
|
|
`dist-browser`, `runner/target`, `provider-assets` (written by the build), `node_modules` and dot-directories; every
|
|
file under `packages/paperclip-eval-kernel` except `dist` and `node_modules`;
|
|
every file under the root `patches` directory, which the acpx profile check
|
|
hashes and the provider entrypoint bundler copies into `dist`; and the root
|
|
`package.json`, `pnpm-lock.yaml`, `pnpm-workspace.yaml` and
|
|
`tsconfig.base.json`). `--finish`, its last step, recomputes the digest and
|
|
writes the stamp
|
|
`packages/paperclip-runner/node_modules/.cache/paperclip-runner-build-inputs.json`
|
|
only if the inputs are still the ones the build began with, together with the
|
|
path and size of every file under `dist`. An edit that lands while the build
|
|
runs leaves no stamp, so the next consumer rebuilds.
|
|
`node server/scripts/prepare-runner-vendor.mjs` reuses the build when the
|
|
stamp is for this platform, every recorded output is still there at its
|
|
recorded size, and the current digest matches. Anything else (no stamp, a
|
|
changed input, a deleted or truncated output, a stamp from another platform)
|
|
runs `pnpm --filter @paperclipai/paperclip-runner build` exactly as before, so
|
|
a standalone `pnpm --filter @paperclipai/server build` on a fresh checkout
|
|
still builds the Runner itself. Inside `pnpm build` the Runner has just been
|
|
built by the same command, and the step skips a 12-16s rebuild. The step
|
|
prints one `paperclip.runner-vendor.timing.v1` line saying which it did.
|
|
|
|
Set `PAPERCLIP_RUNNER_VENDOR_REBUILD=1` to rebuild regardless of the stamp. A
|
|
stamp is only written after a Runner build that passed every step, and both
|
|
the Runner's first build step and the prepare step remove any existing stamp
|
|
before a rebuild starts, so a build that stops partway never leaves a stamp
|
|
that vouches for it.
|
|
|
|
Set `PAPERCLIP_RUNNER_VENDOR_REQUIRE_CURRENT=1` when the Runner under
|
|
`packages/paperclip-runner/dist` was built elsewhere and this build must not
|
|
build it again, for example because it has no Rust toolchain. The step then
|
|
reuses a current build as usual. When the build is not current it prints the
|
|
reason and fails, without starting a rebuild and without removing the stamp.
|
|
Setting both variables fails. The `build` stage of the `Dockerfile` sets it:
|
|
that stage copies `dist` and the stamp out of the `runner-artifacts` stage
|
|
(see "Runner image" in `doc/DOCKER.md`).
|
|
|
|
## Task search evaluation
|
|
|
|
The task search relevance rubric and regression corpus are documented in
|
|
[SEARCH.md](SEARCH.md). Run the real PostgreSQL relevance suite with:
|
|
|
|
```sh
|
|
pnpm exec vitest run server/src/__tests__/task-search-quality.test.ts
|
|
```
|
|
|
|
Set `SEARCH_EVAL_SCALE=1` to additionally measure a disposable 10,000-task,
|
|
30,000-comment dataset. `SEARCH_EVAL_REPORT=/tmp/search-quality.json` saves
|
|
per-query results and latency measurements; scale measurements are opt-in.
|
|
|
|
## Native skill preparation benchmark
|
|
|
|
Measure native skill preparation through the production service, real PostgreSQL,
|
|
and an in-memory OpenTelemetry exporter:
|
|
|
|
```sh
|
|
pnpm --filter @paperclipai/server exec tsx scripts/benchmark-native-skill-preparation.ts
|
|
```
|
|
|
|
The default fixture has a 208-file repository of about 10.8 MB. The report includes
|
|
cold and warm resolution, a fresh service instance, detailed phase timings, and
|
|
regular-pool SQL statement counts (excluding the separate cache-lock connection).
|
|
Every delivered file is checked outside the timed section. No provider is started.
|
|
`--snapshot-file=/absolute/path/snapshot.json --entrypoint=skills/example/SKILL.md`
|
|
uses a captured repository file inventory instead. `--network-delay-ms=25` adds
|
|
25 ms in each direction through a local TCP relay; this is simulated transport
|
|
latency, not a measurement of a hosted database.
|
|
|
|
`--actor=agent` adds a real agent, run, and responsible user, then records version
|
|
pins through the same callback boundary as execution. `--staging-shape=true`
|
|
seeds 14 projects, 10 sources, 8 bindings, and 156 other skills. These match one
|
|
observed staging inventory's counts; their content is synthetic. Combine these
|
|
options to include authorization and unselected library rows in the measurement.
|
|
|
|
By default, spans stay in memory. `--export-otlp=true` also exports them using
|
|
the operator's `OTEL_EXPORTER_OTLP_ENDPOINT`, protocol, and headers. Keep header
|
|
values out of command arguments and reports. The report records source hashes,
|
|
the export mode, and trace IDs. Exported traces identify this local diagnostic
|
|
in their service version; they are not evidence of staging performance.
|
|
|
|
If creating another embedded cluster is impractical, `--postgres-port=55448
|
|
--postgres-user=paperclip` uses an existing **local** server. It creates and migrates
|
|
a uniquely named disposable database and drops that database afterward; it does
|
|
not reuse an application database.
|
|
|
|
## Repository workspace admission benchmark
|
|
|
|
Measure a fresh checkout with and without the repository object cache:
|
|
|
|
```sh
|
|
pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-repository-object-cache.ts
|
|
```
|
|
|
|
This uses a synthetic 32 MiB repository and the real `file://` Git transport.
|
|
It reports cache priming separately, then alternates five fresh checkouts per
|
|
mode. Git's pack trace records received bytes. Each checkout has independent
|
|
objects and passes `git fsck`. There is no injected latency or hosted network
|
|
measurement.
|
|
|
|
Measure cold sandbox staging with a disposable 10,000-file Git repository:
|
|
|
|
```sh
|
|
pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-workspace-staging.ts --baseline-ref=origin/master
|
|
pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-workspace-staging.ts --baseline-ref=origin/master --transfer-delay-ms=3000
|
|
```
|
|
|
|
The benchmark alternates the baseline and current implementations. Both pack,
|
|
verify, copy and extract real archives, then compare the resulting tree and Git
|
|
HEAD. It retains dirty, deleted and untracked-file semantics and excludes ignored
|
|
files. The transport is a local copy. The optional delay models a slow transfer;
|
|
it does not measure hosted network or agent latency. The report contains every
|
|
sample, whole-staging wall time, archive bytes and phase durations. Seed
|
|
verification and upload can overlap, so do not sum their durations. The script
|
|
temporarily imports baseline source beside the current source and removes it on
|
|
exit. Run without other repository edits or heavy I/O for a useful comparison.
|
|
|
|
Measure the real repository preparation service with a disposable embedded
|
|
PostgreSQL database and synthetic history, without invoking an agent:
|
|
|
|
```sh
|
|
pnpm perf:workspace-admission
|
|
pnpm perf:workspace-admission -- --realpath-delay-ms 2.3
|
|
pnpm perf:workspace-admission -- --distinct-cwds --realpath-delay-ms 2.3
|
|
```
|
|
|
|
Defaults model 8,516 workspaces, 27,971 provenance records and 28 retained
|
|
readers on the shared root. The first command measures ordinary local I/O;
|
|
the others inject filesystem metadata latency for sensitivity comparisons.
|
|
Injected latency does not establish the actual latency of a hosted volume.
|
|
`--distinct-cwds` creates separate folders to exercise bounded concurrent I/O.
|
|
`--service /absolute/path/to/baseline/server/src/services/execution-workspace-repositories.ts`
|
|
compares another checkout, which needs its workspace dependencies installed.
|
|
The JSON report times only the service call, counts filesystem lookups and SQL
|
|
statements, and verifies retained source identities. Setup and cleanup are
|
|
excluded. Source tasks and permissions are synthetic; this is not an end-to-end
|
|
agent latency measurement.
|
|
|
|
## OpenClaw Join Smoke Test
|
|
|
|
Run the end-to-end OpenClaw join smoke harness:
|
|
|
|
```sh
|
|
pnpm smoke:openclaw-join
|
|
```
|
|
|
|
What it validates:
|
|
|
|
- invite creation for agent-only join
|
|
- agent join request using `adapterType=openclaw_gateway`
|
|
- board approval + one-time API key claim semantics
|
|
- callback delivery on wakeup to a dockerized OpenClaw-style webhook receiver
|
|
|
|
Required permissions:
|
|
|
|
- This script performs board-governed actions (create invite, approve join, wakeup another agent).
|
|
- In authenticated mode, run with board auth via `PAPERCLIP_AUTH_HEADER` or `PAPERCLIP_COOKIE`.
|
|
|
|
Optional auth flags (for authenticated mode):
|
|
|
|
- `PAPERCLIP_AUTH_HEADER` (for example `Bearer ...`)
|
|
- `PAPERCLIP_COOKIE` (session cookie header value)
|
|
|
|
## PostHog MCP Live Smoke Test
|
|
|
|
The PostHog smoke targets an already-running authenticated Paperclip instance.
|
|
It is deliberately separate from `pnpm test` and `pnpm test:e2e` because it
|
|
uses a live vendor OAuth flow and creates a short-lived connection plus one
|
|
fresh-run proof issue.
|
|
|
|
```sh
|
|
INTEGRATIONS_POSTHOG_PAPERCLIP_E2E_EMAIL=operator@example.test \
|
|
INTEGRATIONS_POSTHOG_PAPERCLIP_DEV_LOGIN_PASSWORD='<environment-delivered>' \
|
|
INTEGRATIONS_POSTHOG_POSTHOG_PROJECT_ID=483530 \
|
|
pnpm smoke:posthog-live https://paperclip.example.test
|
|
```
|
|
|
|
The command fails before browser launch unless all three integration bindings
|
|
are present, and never prints their values. A Paperclip heartbeat derives the
|
|
target origin from its injected `PAPERCLIP_API_URL`; the positional URL (or
|
|
`--base-url <url>`) selects the running instance for a manual invocation. It
|
|
creates no trace, video, or HAR, begins
|
|
screenshots only after OAuth returns to Paperclip, enables read actions only,
|
|
installs the connection on `CodexCoderPro` only, runs `project-get` with `{}`
|
|
from the board Test panel and a fresh agent run, then removes the connection.
|
|
Sanitized JSON and PNG evidence defaults to `PAPERCLIP_RUN_SCRATCH_DIR` when the
|
|
command runs in a heartbeat; set `POSTHOG_EVIDENCE_DIR` for a different output
|
|
directory.
|
|
|
|
Run the focused harness checks without contacting Paperclip or PostHog:
|
|
|
|
```sh
|
|
node --test scripts/smoke/posthog-live.test.mjs
|
|
```
|
|
|
|
## OpenClaw Docker UI One-Command Script
|
|
|
|
To boot OpenClaw in Docker and print a host-browser dashboard URL in one command:
|
|
|
|
```sh
|
|
pnpm smoke:openclaw-docker-ui
|
|
```
|
|
|
|
This script lives at `scripts/smoke/openclaw-docker-ui.sh` and automates clone/build/config/start for Compose-based local OpenClaw UI testing.
|
|
|
|
Pairing behavior for this smoke script:
|
|
|
|
- default `OPENCLAW_DISABLE_DEVICE_AUTH=1` (no Control UI pairing prompt for local smoke; no extra pairing env vars required)
|
|
- set `OPENCLAW_DISABLE_DEVICE_AUTH=0` to require standard device pairing
|
|
|
|
Model behavior for this smoke script:
|
|
|
|
- defaults to OpenAI models (`openai/gpt-5.2` + OpenAI fallback) so it does not require Anthropic auth by default
|
|
|
|
State behavior for this smoke script:
|
|
|
|
- defaults to isolated config dir `~/.openclaw-paperclip-smoke`
|
|
- resets smoke agent state each run by default (`OPENCLAW_RESET_STATE=1`) to avoid stale provider/auth drift
|
|
|
|
Networking behavior for this smoke script:
|
|
|
|
- auto-detects and prints a Paperclip host URL reachable from inside OpenClaw Docker
|
|
- default container-side host alias is `host.docker.internal` (override with `PAPERCLIP_HOST_FROM_CONTAINER` / `PAPERCLIP_HOST_PORT`)
|
|
- if Paperclip rejects container hostnames in authenticated/private mode, allow `host.docker.internal` via `npx paperclipai allowed-hostname host.docker.internal` and restart Paperclip
|
|
|
|
## Sidebar badge run-query benchmark
|
|
|
|
The badge query keeps the newest 200 runs of nonterminated same-company agents.
|
|
It selects each agent's first observed run before status or Mine filters. An
|
|
observation ordinal preserves equal-time input order. Per-agent ranking orders
|
|
by full creation time first, then that ordinal for exact ties. SQL returns only failure
|
|
IDs, issue bindings, and creation times. Dismissals and fresh issue authorization
|
|
still run in the service.
|
|
|
|
Run the synthetic PostgreSQL comparison with a pinned pre-change revision:
|
|
|
|
```sh
|
|
PAPERCLIP_LOG_LEVEL=error SIDEBAR_REQUIRE_BYTE_REDUCTION=1 pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-sidebar-badges.ts --baseline-ref=bf3ef12f3a78d0b15a32ad57f7e16520fbf6cd5e
|
|
```
|
|
|
|
The harness starts disposable embedded PostgreSQL by default. To use an existing
|
|
isolated test database with current migrations, set
|
|
`PAPERCLIP_SIDEBAR_TEST_DATABASE_URL` to a dedicated database whose name ends in
|
|
`_test`. The harness removes only its own synthetic company fixtures. Do not use
|
|
a development or hosted company database.
|
|
|
|
The report includes service-generated SQL, complete badge equality, candidate
|
|
bindings, alternating warm samples, plans, and UTF-8 response-value bytes before
|
|
Drizzle decoding. This byte count excludes protocol framing. The mixed fixture
|
|
covers shared agents, superseding successes, timestamp ties, Mine users,
|
|
dismissal equality, private and deleted issues, and more than 200 runs. The
|
|
all-failed, all-succeeded, and empty controls expose neutral or slower paths.
|
|
Timings describe local SQL work and do not establish hosted request latency.
|
|
|
|
## Dedicated native recovery test database
|
|
|
|
The native idle continuation fixture reads `PAPERCLIP_RECOVERY_TEST_DATABASE_URL`
|
|
and passes it through `nativeRestartFixture({ dedicatedDatabaseUrl })`. The URL
|
|
must name a dedicated PostgreSQL database ending in `_test`, without database-name
|
|
or connection-option overrides. This explicit option applies pending migrations
|
|
and closes registered clients at cleanup; it does not create or drop that
|
|
database. Its provisioning owner remains responsible for disposal. The environment
|
|
variable alone does not opt other `startRecoveryPostgres()` callers into shared
|
|
storage: they retain their existing fresh-database allocation and cleanup.
|
|
|
|
The native idle-sleep continuation integration uses a controlled local provider
|
|
with the real heartbeat queue, authorized comment wake, native session
|
|
checkpoint, files, retained secret encryption and fresh resolution, and owned
|
|
idle preflight. Credential resolution through the secret owner does not prove
|
|
production backend credential injection. It uses an explicit two-connection
|
|
main pool, alongside the production auxiliary pools, and verifies the driver's
|
|
actual setting. It does not
|
|
exercise the remote Runner harness backup transport or a live provider.
|
|
|
|
The workspace runtime lifecycle test uses the actual local runtime owner and an
|
|
ephemeral HTTP child. Normal run release must remove the original PID, listener,
|
|
and service registry entry before the same owner starts another useful service
|
|
from the retained workspace files. This is physical local resource retirement
|
|
and next-use coverage; it does not establish global tenant quiet.
|
|
|
|
The plugin idle manager lifecycle test uses a real SDK worker process and a
|
|
controlled host state handler that persists worker requests to a temporary file.
|
|
It checks the actual manifest generation, refusal of work under the owned hold,
|
|
and a second useful write after release. This boundary does not exercise the
|
|
production plugin database loader or plugin jobs. Unsupported manifests, active
|
|
manual resources, and missing completion receipts remain sleep blockers.
|