Jannik/PaperClipAI
1
0 Fork 0
mirror of https://github.com/paperclipai/paperclip.git synced 2026-11-09 07:10:34 +01:00
Code Issues Packages Projects Releases Wiki Activity
6,734 Commits 1,626 Branches 2,382 Tags
c66b4a55efdc5675b5a5146197078b94de6407da
Commit Graph
6 Commits
Author SHA1 Message Date
Nicky LeachandClaude Opus 5.5 11ac90e31f fix(build): typecheck server/src/__tests__ and fix every type error (#18739) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server has about 1,100 test and helper files in
`server/src/__tests__`
> - `server/tsconfig.json` excludes that directory, Vitest does not
check types, and no script runs the tsconfig in that directory
> - So a test can call a function with the wrong arguments and still
pass, and no tool reports it
> - This gap hid 2,168 real type errors, and some of them hid tests that
did not test what they claim
> - This pull request fixes all of these errors and adds the directory
to the server layers from paperclipai/paperclip#18737
> - The benefit is that a type error in a server test now fails the
typecheck, with no baseline and no backlog

## Linked Issues or Issue Description

Refs paperclipai/paperclip#18737, which is merged and added the server
layers. Refs paperclipai/paperclip#18738, which is merged and checks the
test projects outside `pnpm -r`, so it is not a duplicate. Refs
paperclipai/paperclip#18137, which records that this directory has no
check.

**What happened?**

No tool checked the types of 1,074 of the 1,101 files in
`server/src/__tests__`. I searched `package.json`, the workflows, and
all tsconfig files. 27 files are in the program of another tsconfig, and
these 27 have 0 errors. 11 of the 27 are in
`helpers/maintenance/tsconfig.json`. No script ran that project when I
measured this. paperclipai/paperclip#18738 now runs it.

**Expected behavior**

The server typecheck checks these files, and a type error fails the
check.

**Steps to reproduce**

1. On the base commit, run `tsc -p server/src/__tests__/tsconfig.json
--noEmit`. It reports 6,536 errors and needs 12.3 GiB.
2. Add a type error to any file in `server/src/__tests__`.
3. Run `pnpm --filter @paperclipai/server typecheck`. It passes.

**Paperclip version or commit**

I measured the error breakdown below on commit `8dfd632452`, with
TypeScript 7.0.2. The branch now includes master up to
paperclipai/paperclip#18865.

**Deployment mode**

Not applicable. This issue applies to development and CI. No production
source file in the server changes.

### Error breakdown

Each row adds one change to the row above.

| Step | Total errors | Change | Cause |
| --- | --- | --- | --- |
| Config as committed | 6,536 | | |
| Add `src/types/*.d.ts` | 4,447 | -2,089 | Config. The Express request
types were missing. |
| Set `rootDir` to the repository root | 2,436 | -2,011 | Config. |
| Set `allowImportingTsExtensions` | 2,181 | -255 | Config. Vitest
accepts `.ts` imports. |
| Add `ES2024.Promise` to `lib` | 2,138 | -43 | Config. Node 24 has
`Promise.withResolvers`. |
| Set `allowJs` for `.mjs` imports | 2,148 | +10 | Config. It removes 8
`TS7016` errors and shows new real errors. |
| Set `declaration` to false | 2,142 | -6 | Config. Tests emit nothing.
|
| Count only files in `server/src` | 2,058 | -84 | Other packages: 53
Daytona, 29 Runner test, 2 UI. |

So 4,478 of the 6,536 errors (69%) came from config or scope, and 2,058
are real type errors in 314 files. The 29 Runner test errors are also
real, which gives 2,087 errors. Tests that master added after this
measurement had 81 more errors, so this pull request fixes 2,168 errors.

This table classifies the first 2,058 errors.

| Kind of real error | Errors |
| --- | --- |
| Wrong argument or assignment type | 1,023 |
| Stale or partial fixture objects | 525 |
| Property that the type does not have | 317 |
| Missing narrowing of null or unknown | 101 |
| Implicit any | 60 |
| Missing name or module | 16 |
| Other | 16 |

### What the type errors hid

Most errors were stale fixtures and loose mocks. These errors hid real
test problems, and this pull request fixes them:

- **A test that never ran.** `instrumentation.test.ts` required
`sdk-trace-base`, which is not installed. So 3 span tests were skipped.
They now run and pass.
- **Assertions that could not fail.**
- `issue-thread-interactions-service` used a company ID as an issue ID.
- `status-card-evaluation-work` compared `[]` with `[]` in a rollback
check.
- `slack-conversation-lifecycle` read `item.issueId`, which does not
exist.
  - `email-channels` used a permission key that does not exist.
- `heartbeat-project-env` imported `LOW_TRUST_REVIEW_PRESET`, which is
not exported. The value was undefined.
- **Postgres processes that did not stop.** 11 test files called `stop`
on the test database helper. That method does not exist, and the helper
method is `cleanup`.
- **Arguments that the service ignored.**
- `methodKey` is now `connectionMethodKey` in the Daytona, Unikraft,
1Password, and Railway tests.
- 108 calls passed `"owner-user"` to a parameter that expects an options
object.
- 9 chat wizard fixtures passed `managementContext` as `source`, so they
ran `existing_app` and not `manifest`.
  - 2 tests passed a `{ userId }` actor that the function ignores.
- Inserts into `environments` passed a `companyId` column that the table
does not have.
- **Problems in tests that master added during this work.**
- `sandbox-creation-operations` read `database.connectionString` when an
external test database was set. In that mode `database` is undefined, so
the fixture would stop with a TypeError. It now uses the external URL.
- `idle-completed-history` set a Dot operation to `dispatched`. That is
a Muse status, and the Dot type does not have it. The fixture now uses
`admitted`, which is a real in-flight Dot status.

### Possible product problems, not changed

These may be real problems in production code or types. I did not change
them, because this pull request does not change production behavior:

- `decisions-service` `service.create` does not validate the lenient
`cancel_issue_tree` input.
- `createConnection` ignores `input.connectionPurpose`.
- The `createConnection` route schema default stores `{}` for
`transportConfig`.
- `prepareManagedAiRuntime` accepted the invalid value `mode:
"connection"`.
- `createBinding` accepts invalid `projectionClass` strings outside
class 3.
- `ensureManagedProjectWorkspace` forwards an untyped `repoRef` by
spread.
- `routineVariableSchema.label` is optional, but `RoutineVariable.label`
is required.
- `ToolRunContext.projectId` may be too strict.
- `withdrawInteraction` reads `issue.status`, which its type omits.
- `issueService.list` and `reconcileNativeFinalizations` omit fields
that they return at runtime.
- `createGitHubExternalObjectProvider` handles `tokenProvider: null`,
but its type forbids this value.

## What Changed

- Fixed the 2,168 type errors. The diff changes 322 test and helper
files.
- Corrected `server/src/__tests__/tsconfig.json`. Editors and the
typecheck now read the same options and report the same errors.
- Added three `test-tree` layers to
`server/scripts/typecheck-layers.mjs`. They read the server layers as
declarations, and any error fails them. A hash of the path selects the
shard, so a new file does not move other files.
- Added `server/scripts/typecheck-stubs/daytona-sdk.d.ts`. Server tests
reach Daytona plugin source, and the root install does not have
`@daytonaio/sdk`.
- Changed two imports in `ui/src/lib/issue-execution-policy.ts` to use
`.js` extensions, because server tests import this file.
- Made `declarationsConfig` and `--declarations-only` skip the
`test-tree` layers, because these layers emit nothing. Other projects
read only the source layers. Added a test for this.
- Updated `doc/DEVELOPING.md`.

## Verification

Run these commands:

```sh
pnpm --filter @paperclipai/server typecheck
pnpm exec vitest run server/scripts/typecheck-layers.test.mjs
```

Results on macOS (arm64), Node 24, 16 cores. Each check ran after the
merge of paperclipai/paperclip#18738 unless the row says otherwise:

| Check | Commit | Result |
| --- | --- | --- |
| Server typecheck, `PAPERCLIP_TYPECHECK_LOW_MEMORY=1`, all 7 layers |
`6e45a08573` | Exit 0, 0 errors, 62 seconds, peak 3.52 GiB |
| `node scripts/typecheck-test-projects.mjs --low-memory` | `a3306bd13e`
| Exit 0, 0 errors, 42 seconds, peak 3.25 GiB |
| The same script with `--reuse-declarations` | `1b878aabc0` | Exit 0, 0
errors |
| `typecheck-layers.test.mjs` and `live-recovery-work.test.ts` |
`01db5634e4` | 90 passed. These include real compiler runs for a pass, a
type error, and an invalid tsconfig. |
| `tsconfig.declarations.json` after a full server typecheck |
`0fb1e4fb0a` | It references only `below`, `cycle`, and `above`. No
generated config names a `tsconfig.layers.json` file. |
| Owned runtime test in `idle-completed-history` | `e8f2f448ac` |
Passed, with the fix that master later replaced |
| `idle-incoming-wake`, `plugin-idle-manager`, and the changed
`chat-channels` test | `6e45a08573` | 30 passed |
| Runner `native-execution.test.ts` | `eab4dcc1f4` | 114 passed |
| The 19 server test files that changed since `eab4dcc1f4`, and
`typecheck-layers.test.mjs` | `259737b700` | 4,275 of 4,277 passed. The
2 failures are in `chat-channels` and are in the list of 7 below. |

The peak is the largest single process, from `/usr/bin/time -l`. Peaks
for the server typecheck were between 3.42 GiB and 3.56 GiB across 8
runs. Peaks for the test projects were 3.25 GiB and 3.37 GiB across 2
runs. I did not measure a cgroup.

**Runtime check of the fixes.** Types do not prove that a test still
works. So I ran all 309 changed server test files, 11,929 tests, before
the rebase onto paperclipai/paperclip#18737:

| Result | Tests | Detail |
| --- | --- | --- |
| Passed | 11,913 | Includes the 2 tests and the 54 `dot-runner` tests
that passed in a later run, as described below. |
| Failed | 7 | The same 7 tests also fail on the commit before the
fixes. |
| Skipped | 9 | These tests have their own skip conditions: `sentry` 5,
and 1 each in `bundled-plugin-registration`, `customer-success-cloud`,
`onepassword-connection`, and `workspace-runtime-https-live-exercise`. |

The 7 tests that fail with and without this change, on my machine:

- 3 tests in `chat-channels` and `email-channels` stop with `ENOENT` on
a `SKILL.md` file. The path is in the parent checkout, above the git
worktree that I used, so I expect that the worktree is the cause. I did
not prove this.
- `workspace-instance-cleanup` compares `/private/var/...` with
`/var/...`. This is a macOS symlink difference.
- 2 tests in `git-reproducible-managed-checkouts` get
`unknown_base_git_files` and expect `unpushed_managed_checkout`. I did
not find the cause.
- 1 more test in `chat-channels`, `materializes direct external-chat
finals once`, gets `undefined`. I did not find the cause.

CI is the check for these 7 tests. `ci / verify` passed on `eab4dcc1f4`.

Failures during this work:

- My fix to `agent-runtime-connector-access` changed an invalid
permission key in the setup, but not in the assertion. The runtime check
found this. I corrected the assertion, and the file passes.
- `heartbeat-plugin-environment` failed one wait in the full run. It
passed three times alone, so I think that load caused the failure. It
can fail again under load.
- `dot-runner` skipped all 54 tests in the full run because `cargo` was
not on my `PATH`. It then passed 54 of 54 with the toolchain on the
`PATH`.
- My first `lib` value removed the default DOM types, and the error
count went up by 375. I added the DOM libraries.
- My Daytona stub with bare `any` left 48 errors, because generic
helpers then inferred `unknown`. I added shapes that return promises.
- One fix to `agent-secrets-routes` used `class_3_static_lease`, and the
test failed. That class allows only specific paths. I reverted to
`unclassified`.
- `ci / General tests (server (8/12))` failed on `259737b700` because of
the flaky port in `idle-completed-history`. paperclipai/paperclip#18853
fixed this test on master, so this pull request keeps that fix.

**Audit of the diff.** I counted these patterns in added and removed
lines, to find fixes that only hide an error:

| Pattern | Added | Removed |
| --- | --- | --- |
| `@ts-ignore` | 0 | 0 |
| `@ts-expect-error` | 0 | 1 |
| `.skip` | 0 | 0 |
| `as any` | 9 | 9 |
| `: any` | 11 | 2 |
| `as never` | 8 | 2 |
| `as unknown as` | 35 | 4 |
| `Promise<unknown>` | 16 | 5 |
| `expect(` | 128 | 113 |

No hunk removes more `expect(` calls than it adds. Three removals of
test content were intentional, and I checked each one:

- `muse-control` removed `signalToken`. Line 1012 already asserts that
it is absent.
- `merged-pr-confirmation-sweep` removed legacy aliases. Other tests
cover them.
- `heartbeat-workspace-branch-containment` removed a `finalize` branch
that no test row can reach.

Some partial fixtures stay unchecked through `as unknown as` and
`Promise<unknown>`, mostly in `issue-agent-mutation-ownership-routes`
and `issue-comment-reopen-routes`.

**Accuracy of the check.** One program over the tree and the three
shards reported the same errors at the same positions. Memory, as the
peak of the largest process on one thread, was 3.93 GiB for one program
and 2.43 GiB for the largest shard. I measured these two values before
the fixes.

Not run or not proven:

- I did not run the full `pnpm test:run`, or a root `pnpm build`. I
expect CI to run the full suites.
- I did not run the 4 GiB container check, or the full
`scripts/typecheck-low-memory.mjs`. The peaks above come from macOS, so
they are not a Linux cgroup measurement.
- I did not run the UI typecheck after the merges.
- `workspace-runtime-https-live-exercise` is skipped, so its `cleanup`
fix did not run.

## Risks

- **A large diff in test files.** 328 files change. A pull request that
edits the same tests can get merge conflicts. New tests on master can
also add type errors before this merges. Any such error fails this check
and needs a fix here.
- **A higher cost for changes to source types.** A change to a type can
add errors in test files that the author did not touch. The author must
then fix them. This is the purpose of the check, but it is new work.
- **Changed test behavior.** Some tests now run paths that they did not
run before, such as the 9 chat wizard fixtures and the 3 span tests.
They pass here, but they can be more sensitive to timing or to the
environment.
- **The Daytona stub can go stale.** The stub is written by hand. If the
plugin uses a new SDK member, the typecheck reports an error until
someone updates the stub.
- **The release path.** `release-verify.yml` and `scripts/release.sh`
run `pnpm -r typecheck`, so they now run this check.
- **The tsconfig change.** The test transform reads
`server/src/__tests__/tsconfig.json`. The options that I changed are for
type checks, and the runs above passed.
- **A gap that remains.** Tsconfigs in `packages/paperclip-runner` and
in most sandbox provider plugins exclude `*.test.ts`. This check does
not read most of those test files, so they can still drift. That work is
separate from this pull request.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

Anthropic Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with
reasoning, tool use, code execution, and subagents. The runtime does not
show the context window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [ ] I have run tests locally and they pass (7 tests fail locally, with
and without this change. See Verification.)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-03 15:27:38 -08:00
DottaandPaperclip ab51cabbbd test: await async outcomes and drive saved deadlines deterministically (#18416) 
Reviewed and verified at ab0b15d234.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-31 08:18:15 -05:00
DottaandPaperclip 100b364264 Preserve selected Google services when setup resumes (#18048) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Connections control which external services an agent can use.
> - Google Workspace setup lets a person choose services and read or
write access.
> - A draft lost that choice after a declined or expired OAuth handoff.
> - Finish setup could then show all services with write access instead
of the saved narrow choice.
> - This pull request saves the exact draft choice without treating it
as granted access.
> - The benefit is safe setup recovery that preserves the person's
choice and existing account limits.

## Linked Issues or Issue Description

Refs #15545. The duplicate search found no open PR for Workspace draft
consent recovery. Related #18000 concerns satisfied gateway access
requests, not saved OAuth setup intent.

**What happened?**

A cold Workspace draft resumed with the new-account service defaults.
The selected request lived only in short-lived OAuth state. Denial
consumed that state. The picker ignored draft connections.

**Expected behavior**

Finish setup restores the same person's saved service choice. It does
not use another person's choice, a broad profile hint, or a credential
grant as draft intent. An old draft with no reliable choice asks the
person to choose services.

**Steps to reproduce**

1. Start a new Google Workspace connection. Select Gmail read-only. Turn
off Drive and Calendar.
2. Continue to Google, then decline authorization.
3. Close the setup page. Return through Finish setup.
4. Open the service picker. Before this fix, all three services can be
enabled with write access.

**Paperclip version or commit**

Reproduced in tests on master
`820c458465fa40137c81fe510f3e6dc573b4f895`.

**Deployment mode**

Authenticated and local source builds. The source regression uses real
PostgreSQL fixtures and the production React setup components. Live
provider acceptance of this repair is still pending deployment.

## What Changed

- Save validated, non-secret draft intent before managed or
customer-owned OAuth. Bind it to company, connection, actor, grant kind,
and subject.
- Restore that intent after denial, state expiry, and cold Finish setup.
Keep active reconnect bound to its saved grant. Do not change
new-account defaults.
- Preserve agent-to-human setup handoffs. Consume only the addressed
person's reviewed proposal. Keep other people's proposals.
- Lock and merge the current draft before saving intent. Return 409 if
its credential identity or method changed during setup.
- Preserve the latest saved choice across overlapping customer-owned
starts. Merge both current configuration records under short locks.
Reject stale client-binding changes. Keep each OAuth state's own
requested scopes.
- Add shared, service, handoff, picker, and cold-page regression tests.
Update the connector contract and existing feature plan.

- Route the default ungrouped test/release paths through six sequential
exact-cover chat partitions. Preserve remaining server/native coverage,
explicit grouped CI, selection-first execution and shared-database
paths.
- Correct the retention test driver so pending owned real I/O does not
advance logical deadlines. Pump zero-duration connection timers and add
a held-I/O regression; keep production scheduling and original timing
assertions unchanged.

The shared helper is needed by the server and picker to validate the
same narrow binding. It does not resolve credentials or grant access. No
production schema migration, new route, provider scope or deployment
configuration is added. The test-harness changes do not change
production timers.

## Verification

### Current candidate: source qualification passed; live acceptance
pending

- Current published candidate:
`82d13d4d698a5579378b5d506cb7a1f9cb64b5c8`, tree
`78d6a821ce53f481fe3e9991edce56f0acffb22f`. It combines the Workspace
repair with launcher commit `bf29c9` and retention-test commit `e25987`,
then integrates master `493e229d93ba7798e80152f4549e2e9177ce50ae`. The
integration brings 40 upstream commits across 713 files. One launcher
conflict keeps upstream validated CI selection and its selection-first
path, then applies six default chat partitions only without a selection.
The authored PR diff has 16 files, 833 additions and 100 deletions; it
adds no authored lockfile or `.github` changes. The public head is
confirmed at this revision. Fresh CI and review passed as recorded
below. Source qualification is complete using exact-tree full CI and the
local results below. The separate local whole command was interrupted by
host disk protection and is not called a pass. Live repaired-draft
acceptance remains pending deployment.
- Exact clean `82d13d4` passed one Node 24.19.0 builtin-only run: 41/41
cases (37 launcher and four CI-workflow cases), zero failures/skips.
HEAD, tree and relevant file hashes stayed unchanged and clean before,
at start and after. This proves launcher fixtures, not real chat
execution, current typecheck/build, the dependency-backed selection
graph, or full feature acceptance.
- The new default ungrouped whole command uses six sequential
exact-cover chat partitions, then the remaining server suites,
workspaces and serialized suites: 28 expected invocations. It preserves
all native suites, including upstream usage coverage. The plain
ungrouped release launch path also changes. Explicit grouped CI,
validated selection and shared-database execution stay unchanged.
Launcher hard RED had three expected body failures and two unchanged
passes; the repaired focused five and final 37 passed. A later 41-case
staged-merge pass is retained separately; the clean 82 run above is the
frozen proof.
- Fresh [CI run
41153113944](https://github.com/paperclipai/paperclip/actions/runs/41153113944)
completed successfully on October 31 at 01:14:53 UTC. Current head has
52 successful check runs, four conditional skips and no pending/failing
checks; legacy Snyk also passed. The skips are test-file selection, the
optional paired benchmark and two Storybook checks. The full
server/chat/workspace/serialized matrix, all eight e2e shards, normal
build, build-gap typecheck, Runner checks and final aggregate gates
passed. Build and all three chat job logs identify checkout
`e5e831fd3a8534d0d0f1e892118c0f8fbe12dd92`, merging this head into
`493e229d93`. Its Git tree is exactly
`78d6a821ce53f481fe3e9991edce56f0acffb22f`, identical to the clean
candidate. The six real chat partitions passed 314, 358, 273, 239, 321
and 311 cases: 1,816 total. Other cases shown as skipped in each
partition were selection exclusions, not extra passed cases. This is
current tree-equivalent CI proof, not a pass of the separate default
ungrouped local whole command.
- Fresh [Greptile
summary](https://github.com/paperclipai/paperclip/pull/18048#issuecomment-6473856992),
edited October 31 at 01:05:04 UTC, explicitly reviews exact head
`82d13d4` at 5/5 with no actionable code issue. Its exact-head check
passed. The prior overlapping-start discussion is resolved. This review
is source-only, not provider or live acceptance.
- Exact clean `82d13d4` passed full recursive typecheck and normal `pnpm
build` on the independent darwin/arm64 snapshot. Typecheck finished on
October 31 at 01:27:33 UTC; build finished at 01:28:33 UTC. Both exited
0. Normal native preparation and code signing passed. Source HEAD, tree
and tracked/untracked state stayed unchanged and clean. `pnpm
check:token-gates` also passed all token, palette and ClipLab checks on
this candidate. The earlier forced-clone `ENOSYS` preflight remains
recorded as setup failure, not an assertion result.
- The unchanged default `pnpm test:run` started at 01:51:33.796 UTC on
clean exact `82d13d4`. All six real chat partitions passed: 358, 314,
273, 239, 311 and 321 cases, or 1,816 actual cases. The original FIFO
and ambiguous-retry cases both passed under their unchanged 15-second
limits. General-server then reported 37 completed file rows with 5,438
passed cases and five default remote-continuity skips, including all 498
tool-access service cases and all 83 connection-route cases. At 02:39:44
UTC, the unchanged 5-GiB host-disk guard terminated the owned run. The
child ended with SIGTERM and the container/harness exited 143, with no
OOM or recorded assertion failure. Only seven of 28 RUN invocations
started. Retention had no completed local row. Later invocations did not
run. This whole local command is interrupted/incomplete, not passed.
- A coverage check found no default suite missing from current full PR
CI. The maintained launcher tests prove exact coverage for all 12 PR
server shards, chat and five native suites; all configured workspace
projects; and all nine serialized shards. The final Runner CI shard
executes the five native server suites. The 41 local launcher/workflow
cases cover six sequential chat children, per-child isolation, native
retention, fail-fast propagation and unchanged explicit
selection/shared-database paths. The actual six-chat run above exercises
that changed default path. The remaining unqualified local difference is
uninterrupted sequential whole-command composition and resource
accumulation, not omitted test selection. Current exact-tree full CI
supplies broad source-test qualification; no local whole-pass claim or
provider acceptance is inferred.
- A task-owned remote runner was considered to complete that separate
composition check. Its provider rejected the requested allocation with
per-sandbox limits of 8 GiB RAM and 10 GiB disk. No sandbox was admitted
and no remote tests ran. No tests, timeouts, assertions, database
synchronization, source or provider settings were changed to avoid the
capacity limit. Historical a0 failures remain below. No deployment or
live repaired-draft acceptance has run.

### Bounded follow-up on the prior source

- On a0, one finite phase-traced chat diagnostic executed exactly the
two failed cases. Both passed (1,374 and 1,767 ms); 1,814 other cases
were filtered, not passed. This did not reproduce a blocked await or
establish the original monolithic timeout cause. No assertion or
15-second limit changed.
- The retention diagnosis preserved an initial zero-body selector
failure, then corrected only the selector: one original case passed in
783 ms, with 42 filtered. Trace showed the old settling helper advancing
logical time while real I/O was active. A new held-I/O regression then
failed hard because the old helper advanced 1,000 ms. This proves a
test-driver defect, not the precise cause of the old 5,340 ms full-run
gap.
- The first manual-clock repair failed the complete 44-case file: 43
passed, one body hit 15 seconds and cleanup hit 30 seconds. Its planned
mutant did not run. Source then identified a fresh PostgreSQL
connection's zero-duration timer. The corrected driver pumps
zero-duration timers while initial readiness or tracked I/O is pending,
without advancing logical Date. That pre-integration file passed 44/44
with no skips. A separate scratch-only 5,000 ms normal-continuation
mutant failed the original hard `5000 < 3000` assertion; its expected
failure is not a test pass. The original page-count, fairness, expiry
and quiet assertions stayed intact. The authored repair does not change
the production worker. The merged master changes an older retention case
and dependencies; these pre-integration results are not a complete 82
retention or whole-suite pass.

### Historical qualification: SHA-bound, not 82 acceptance

- Historical a0 candidate: `a0f9089fc1ff466bf4e1eeddc5b1a910d09e380b`,
tree `b9d73713319dcd52b9eaa6053008b5190e43fdf1`. It adds the
overlapping-start repair to parent `6c18`, which already merged pinned
master `1511c92965325249ed6b52dd5be52eb6d736fbfa`. That candidate
changed 12 files. Its CI and review passed; its local full qualification
remained failed/incomplete.
- Greptile's previous `6c18` review found P1 [Latest service choice gets
lost](https://github.com/paperclipai/paperclip/pull/18048#discussion_r4425965828),
confidence 4/5. Two hard-assert regressions reproduced it on untouched
`6c18`: the already-bound path overwrote the later choice, and real
first-use client stamping also restored the old choice. Existing OAuth
states/grants were not widened, but a later setup could restore an older
request. The new repair covers both write boundaries without holding
locks over external work or copying another attempt's scopes.
- The complete affected service file passed 498 tests with no failures
or skips. All 31 required controls ran, including both unchanged RED
cases, 14 new client-drift cases, seven earlier identity guards,
reconnect/cold-resume cases and upstream transaction tests. The tested
three-file patch is byte-identical to this commit; the connector
contract update is separate. Independent scoped review found no
additional actionable issue. Fresh Greptile reviewed exact `a0f` at 5/5
with no actionable findings. The previous P1 thread is resolved.
- On exact `a0f`, the five shared/UI files passed 312 tests. An initial
complete connection-management run then failed: 74 passed, eight timed
out at the unchanged 15-second limit, and one AI-account verification
returned 422 instead of 201. The Google addressed-user handoff passed.
The container exited 1 without OOM and stopped before chat/token/whole
tests. This material failure remains retained and its cause
unclassified; the later controlled recovery below is not a causal
explanation.
- One diagnostic executed the first failed gateway case and the two AI
cases with temporary phase-only tracing. All three passed at 8.516,
5.511 and 3.523 seconds; 80 other cases were filtered. Original hard
assertions and timeouts stayed unchanged. No timeout or suspected late
provider cleanup reproduced, so the complete-file failure remains
unresolved. Vitest exited 0; the harness exited 1 because its trace
parser missed ANSI prefixes. All 203 events were recovered from retained
logs without rerunning tests. Instrumentation was removed. This is not
acceptance.
- Historical a0 CI
[41097596364](https://github.com/paperclipai/paperclip/actions/runs/41097596364)
completed successfully: 52 successful checks, two expected Storybook
skips and no failed/pending checks. Legacy Snyk passed. All 48 CI jobs
passed, including all 1,816 chat cases and explicit passing lines for
the original 20 local failures. [Server
11/12](https://github.com/paperclipai/paperclip/actions/runs/41097596364/job/124207469276)
explicitly passed the complete 83-case connection file in 129.49
seconds, including all nine local failure titles. The actual checkout
was synthetic merge `06851da6eb969d63e1c7dc9f7c95f08d09fa4ff1`,
combining `a0f` with `aa58fa107389e8b9ae5949f7ad815943dfd94292`. This is
not standalone local `a0f` qualification and does not erase the nine
local route failures.
- Linux admission uses Node 24.22.2, pnpm 9.15.4 and Rust 1.97.1. Frozen
offline install, exact native hashes and real migrated PostgreSQL 18.1
passed. Native/protocol/build/lock inputs are identical between `6c18`
and `a0f`; validated cache reuse retains the actual `6c18` native-build
provenance. The selected packaged release binary is not called a fresh
`a0f` build. Database synchronization settings stayed on. This admission
alone does not explain the route failures.
- A fresh Mac snapshot of exact `a0f` passed frozen install, workspace
prerequisites, full sequential recursive typecheck and normal `pnpm
build`, both on their first attempt. Typecheck completed at 17:43:05
UTC, build at 17:44:43 UTC; both exited 0 with clean tracked/untracked
source. Pinned Node 24.19, pnpm 9.15.4, Rust 1.97.1, package concurrency
one, Cargo jobs one and a 12 GiB V8 heap ceiling were retained.
Largest-process maximum RSS was 7,668,154,368 bytes for typecheck and
10,038,837,248 bytes for build, not aggregate tree peaks. The first
build correctly withheld its freshness stamp because 125 ignored, pinned
Copilot assets appeared during preparation; exact path/hash analysis
explained this. The normal build then recorded a genuine stamp and
reused Cargo outputs in 1.18 seconds. No manual stamp, source workaround
or gate retry. This is darwin/arm64 static/build proof, not Linux
packaging/runtime or full tests.
- The historical a0 candidate's token checker passed all four gates
across 1,410 files, run directly with pinned Node 24 against the script
behind `pnpm check:token-gates`.
- The single controlled complete-file comparison passed: baseline `6c18`
83/83 in 287.07 seconds, then candidate `a0f` 83/83 in 293.03 seconds,
with zero skips. All nine earlier failure titles passed on both. Both
child/container exits were 0, no OOM/interruption; source and
native/resolver/PostgreSQL guards matched. This ran only after all owned
Mac compilation ended, with fresh processes/databases and identical
four-CPU-quota/6-GiB/overlay inputs. No filtering, instrumentation,
assertion/timeout/source changes or retry loop. This establishes
standalone complete-file recovery, not the cause of the earlier eight
timeouts/AI 422.
- The first unchanged full `pnpm test:run` on exact clean `a0f` ran from
October 30 at 18:01:43 UTC to 21:06:19 UTC. It failed naturally with
test and container exit 1, without OOM. General-server reported 1,257
passed, one failed and seven skipped files; 28,633 passed, one failed
and 103 skipped tests. Only the first of 22 invocations ran. The later
four groups and 21 invocations did not run. The sole failure was the
unchanged chat-shard CLI meta-test: its nested projectless pnpm lookup
hit the offline network guard. This was a failed full gate, not Google
provider acceptance.
- A finite environment pair reproduced that one meta-test failure, then
passed the same unchanged case after supported `corepack prepare
pnpm@9.15.4 --activate` as the existing non-root node user. The cached
pnpm payload was identical. Only the missing node-user last-known-good
record changed. Core source, dependencies, lockfile, fixtures,
assertions, timers and offline guards stayed unchanged. This isolates
the nested collection defect; it does not explain the earlier route
failures or complete the full gate.
- The corrected unchanged full command ran from October 30 at
21:29:03.500 UTC to October 31 at 00:02:44.207 UTC on the same exact
clean `a0f`. It failed naturally with exit 1, signal null and no OOM.
Source stayed clean before, at start and after. It retained two visible
CPUs/quota, 6 GiB, private IPC, normal overlay, matching native inputs,
synchronized real PostgreSQL 18.1, and the default five groups/22
invocations. Only the first general-server invocation ran; the other 21
did not run. General-server reported 1,265 files: 1,256 passed, two
failed and seven skipped. It reported 28,737 tests: 28,631 passed, three
failed and 103 skipped. The two chat cases emitted `Test timed out in
15000ms` at `chat-channels.integration.test.ts:77310` and `:81790`, with
durations of 15,045 and 15,058 ms. These were the durable pre-run FIFO
notice and ambiguous GitHub text restart/retry cases. The retention case
emitted `expected 5340 to be less than 3000` at
`skill-version-retention.test.ts:593`. Its real duration was 1,727 ms;
5,340 ms was the recorded fake-time directory-page gap, not a deadline
timeout. The timeout stacks do not identify the blocked internal await.
The cause of these failures remains unclassified. Inside this same
failed run, the complete 83-case connection file and 498-case service
file passed. The former Corepack meta-case also passed in 6,396 ms.
These component passes do not waive the failed/incomplete full gate. No
rerun, cancellation, timeout/assertion change or full-pass claim. A
later fresh two-case phase trace passed both chat cases; it did not
reproduce or establish the original timeout cause. At that checkpoint,
the PR stayed draft.
- Prior `6c18` targeted checks passed 930 tests across eight files and
token gates. Its selected queue group passed all 53 cases, including the
original 20 failures and 12 new named controls. Backlog cases took 10.49
and 8.45 seconds under the unchanged 15-second limit. Another 1,763 chat
cases were intentionally filtered. This is historical targeted recovery,
not a full pass or proof of the new repair. No timeout or assertion was
relaxed.
- Prior `6c18` full local qualification failed during serial typecheck:
server TypeScript was killed with code 137. Docker confirmed OOM in the
6 GiB container; full tests and build never started. The failure is
retained. The new Mac resource plan avoids repeating that undersized run
and does not change the shared Docker VM.
- Prior CI
[41089793558](https://github.com/paperclipai/paperclip/actions/runs/41089793558)
completed successfully for synthetic merge
`a73bffcfa3ede051961bcabcfa576c703d10c932` (`6c18` plus
`abc270ee4c03e13b4d6eaad156a41b4ebed4bb0a`). All 48 CI jobs passed; all
1,816 chat cases passed, including explicit log lines for the original
20. Check runs were 51 success, two expected skips and one Greptile
failure; legacy Snyk passed. These results do not qualify `a0f` or erase
its local route failures.
- Red proof on the unmodified master baseline: cold picker and Finish
setup restore broad defaults; managed and customer-owned drafts lack
saved intent. The stale setup tests also fail before the lock guard.
- Live staging baseline `b25f3e2af8e77d882cd22d36ca55640e6468e46f` also
reproduces the ordinary failure: select Gmail read-only only, cancel at
Google, reload the connector catalog, then Finish setup. All three
groups return enabled at read/write. Authorization was declined; no new
Google access was granted. This is baseline failure evidence, not a
deployed-fix pass.
- Historical affected suites: 800 tests passed before the final lock
guard. The final focused service and handoff cases passed 126 tests
after that guard. These are not results for the integrated candidate.
- Historical `pnpm -r typecheck` and final lock-guard server typecheck
passed. The historical a0 candidate passed the full recursive gate
separately, as recorded above.
- `pnpm build`: passed again on the final lock-guard source in commit
`1c31cde0ed199fbf4e41b7649590554f7886dfc2`.
- Historical `pnpm check:token-gates` passed; the historical a0
candidate's direct checker pass is recorded above.
- `pnpm test:run` on macOS failed on the frozen commit. It exited 1
after 9,432.41 seconds. General server results: 1,195 files passed, 10
failed, eight skipped; 26,706 tests passed, one timed out, 157 skipped.
Nine setup hooks timed out while starting embedded PostgreSQL. The
retained log records host-wide semaphore exhaustion (`semget`), not disk
exhaustion. One unchanged publication-text performance case also timed
out. Its cause is not established. Later workspace and serialized groups
did not run. This is a failed and incomplete local gate.
- The independent full-source Linux run also failed and was incomplete
on the same frozen commit. Native and real-database admission passed.
Chat integration reported 20 failures in 1,593 tests. Tool-access passed
all 479 tests. Native-session-executor passed all 965 tests. Docker
interrupted the run after about 48 minutes on October 29. The container
exited 143 and was not OOM-killed. The run produced no final group
totals or detailed assertion summary. Shell trap metadata reports zero,
but this is not a valid test-pass result. Source and evidence remain
unchanged. The focused diagnosis below retains those failures; their
cause is not assumed from the interruption. Neither historical local
full run passed.
- A fresh focused replay on the frozen source then executed exactly
those 20 case bodies. All 20 failed; this was not a setup-hook failure.
The two backlog cases reached the unchanged 15-second timeout. The 18
FIFO cases failed on missing notices or receipt rows. All 18 FIFO cases
passed when run alone in a fresh process and database. This supports an
order interaction, not a complete cause or baseline conclusion. A
separate two-backlog control used RAM-backed temporary storage and real
PostgreSQL with synchronization settings on. One backlog case passed in
10.65 seconds; the other still timed out at 15 seconds. No source,
assertion or timeout was changed. The full local gate remains open.
- Earlier mixed-revision run: it began before the final guard and
imported older source before collecting later regression cases. It
failed those new cases with that older cached module. This run is
retained, not counted as candidate acceptance. The fresh focused run
passed all seven stale-setup cases. No case was skipped to hide a
failure.
- Historical remote gates for `1c31` were terminal and clean: 53
successful contexts and two expected Storybook skips. Main CI
[40720536514](https://github.com/paperclipai/paperclip/actions/runs/40720536514)
passed. Greptile reviewed exact head
`1c31cde0ed199fbf4e41b7649590554f7886dfc2` at 5/5 with no actionable
findings; the separate review job also passed, and there were no inline
review comments. These results do not qualify `6c18`.
- CI source qualification: the chat jobs checked out synthetic merge
`ef36746b3d2ec52ba5727dcbe4da265a03421fe4`, which merges the PR into
`7d75ae0569e41786640cf09647ae8849411d2f22`. All 20 locally failing case
names have explicit passing CI log lines. These passes do not establish
standalone local acceptance of `1c31`. The merge adds two upstream Teams
tests, so it collects 1,595 chat cases instead of 1,593. Neither
backlog/FIFO assertions nor their timeouts changed.
- Independent source review found an earlier credential-identity race.
The guard and seven deterministic cases resolve that finding. A later
integration review missed the separate overlapping-start race. Both RED
failures and that review miss are retained. The narrow follow-up repair
passed 498 service cases, independent source review and fresh remote
CI/5-of-5 review; full local qualification remains incomplete.
- Source qualification is ready for merge. Live repaired-draft recovery,
delivery and Google approval are not established by these results. The
local whole command remains explicitly incomplete.

## Risks

- Historical drafts without a matching intent now need an explicit
service choice. This is deliberate and prevents silent broadening.
- A concurrent credential or method edit now returns 409. Reloading lets
the person review the changed setup.
- Overlapping starts keep separate requested scope sets. Updating draft
intent does not relabel an earlier OAuth state's consent request.
- Saved intent is not consent. Provider-reported scopes and current
action policies still control discovery and dispatch.
- No migration is needed. No existing token or grant is rewritten by
saving draft intent.

- Default ungrouped launch paths now execute six chat children in
sequence and 28 whole-command invocations. Exact coverage, local
launcher contracts and all six real default chat partitions passed. A
complete uninterrupted local whole-command run remains unqualified
because of host capacity.
- The retention test-clock repair depends on owned-I/O and zero-duration
connection-timer progress. The controlled regression, sensitivity mutant
and current exact-tree CI protect the original assertion. It did not
reach a completed row in the interrupted local whole run.

## Model Used

OpenAI Codex, described by the runtime as GPT-5 based. The exact runtime
model ID and context-window size are not exposed to this task. Used
reasoning, code execution, browser verification, and independent
delegated review.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used, including the runtime metadata
limit
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have described the issue in-PR following the bug report template
- [x] I have not referenced internal or instance-local Paperclip issues
or links
- [x] My branch name describes the change and contains no internal
ticket or instance identifier
- [x] I have run tests locally and they pass — exact clean 82 passed 41
launcher/workflow cases and all 1,816 real default chat cases, full
recursive typecheck, normal build and token gates. Full exact-tree PR CI
passed. The separate local whole `pnpm test:run` was interrupted by disk
protection and remains incomplete; it is not claimed as a local whole
pass.
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented risks above
- [x] All Paperclip CI gates are green — current exact-head checks
passed; the verified CI checkout tree equals clean 82. The separate
interrupted local whole command is disclosed above.
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups —
fresh exact-82 summary and check passed; the earlier finding is
resolved. This is source review, not acceptance.
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-30 22:02:58 -05:00
DottaandPaperclip 0e15c92073 feat(skills): make managed content independent of web-server disk (#18334) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agents use installed skills and exact version pins during execution.
> - Managed skills still depended on mutable directories on the web
server.
> - A replacement server could lose acknowledged edits or remove an
unassigned skill whose directory was absent.
> - This pull request makes immutable versions the authority for managed
content and installed repository reads.
> - The server can rebuild delivery files from durable content after
replacement.
> - This establishes skills readiness for ephemeral web servers. General
server draining and workspace recovery remain separate work.

## Linked Issues or Issue Description

**Subsystem affected**

Skills services, shared contracts, database receipts, Skill Studio, and
runtime delivery.

**Problem or motivation**

A fresh web server must discover, read, edit, and deliver protected
installed skills using the same database, object storage, secrets, and
supported application artifacts. It must not need its predecessor's
files or process memory.

**Proposed solution**

Use `managed` ownership with a null source locator. Read installed
content by immutable version. Prepare packages before a short
publication transaction. Persist execution selections before
materialization. Recover exact pinned runtime assets and reject deletion
while active or recovering pins exist.

**Alternatives considered**

Reconstructing a mutable installed directory preserves the old
source-of-truth ambiguity. Permanent dual writes create two competing
authorities. This change uses one managed model after a bounded legacy
conversion.

**Roadmap alignment**

This improves the existing Skills Manager, Studio, version history, and
Cloud execution capabilities. The maintainer supplied the implementation
contract. This does not add a separate skills product or deploy fleet
changes.

Related work: #18082 supplies immutable package storage. #17273
addresses edits outside the API for linked local skills; this PR keeps
that source category separate. #2648 is a closed earlier
ephemeral-filesystem proposal.

## What Changed

- Add managed ownership and focused content publication and execution
materialization modules. Keep the archive format and package storage
tables.
- Publish managed create, save, delete, fork, rename, restore, and
catalog content with version checks. Add durable create receipts.
Preserve file-save receipt responses.
- Return file version identities. Keep Studio drafts after a conflict
and provide an explicit reload action. A delayed reload cannot overwrite
newer typing or navigation.
- Persist exact runtime selections, including empty selections. Rebuild
pinned assets and verify their admitted digests on a replacement server.
- Preserve source access on forks and disconnected copies. Keep the
existing workspace policy category for managed skills.
- Add bounded conversion, eligibility, and independent verification
phases. Load persisted Cloud configuration in the standalone CLI without
seeding or changing it. Reject partial current versions against
installed inventories; recover only complete saved content or the exact
source. Document storage migration blockers and the compatible rollback
floor.
- Add separate-process PostgreSQL/S3 qualification and update domain,
retention, policy, and UI regressions.
- Pass the plugin worker manager into Studio execution, admit restricted
`skill_test` mode through native Runner contracts, and preserve admitted
scope across native and REST tools. Native tests cannot create ordinary
tasks, browse company data, or use external tools.
- Select the exact durable Studio revision even when unassigned or Beta
skills is off. Persist Studio output and audit atomically with native
task completion, preserving the result on replay.

## Verification

- Final revision `13e8af480b88ba6fea5737bac4bd81104d604792` passed full
CI: all 49 workflow jobs completed successfully ([run
41127396098](https://github.com/paperclipai/paperclip/actions/runs/41127396098)).
This includes typecheck, test suites, Runner checks, browser checks and
build. Greptile review 15 is 5/5 on that same revision, with zero
unresolved threads. The PR is ready for review.
- Local workspace typecheck and build passed after integrating master.
Focused checks passed: 33 execution/completion cases, 52 Runner HTTP
cases, 16 managed-content cases, 17 Studio cases, the new real-process
CLI regression, map checks and UI token gates. The CLI regression proves
inspection does not seed missing Cloud configuration. Server typecheck
passed after that fix.
- The independent-process PostgreSQL/S3 matrix passed through normal
HTTP skill routes. It covers A → B → C replacement, lost commit
responses, upload crashes, concurrent saves, binary contents and
executable delivery. Metadata listing and warm selected delivery retain
zero object downloads. Local timing measurements are recorded; they do
not establish hosted latency.
- Hosted A → B → C passed on the owned disposable test tenant with
external Postgres, S3 and a real remote Paperclip Computer. Predecessor
temporary homes were absent. C read and executed B's acknowledged
managed save. New runs used repository v1 while v2 was installed. Hashes
matched an independently checked exact upstream commit.
- Compatible image rollback to `6518b323` passed. Studio retained the
unassigned fork's C save. A fresh remote run executed the newer managed
content and historical repository pin. The actual shell event confirmed
expected hashes, stdout, exit 0 and mode 0555.
- Runtime candidate `9200e3fcb4` deployed successfully ([Cloud run
41124147876](https://github.com/paperclipai/paperclip-cloud/actions/runs/41124147876)).
The final Studio test delivered only the unassigned fork's exact v3,
executed its script and persisted a successful output snapshot.
Independent command-event inspection confirmed all three hashes, stdout,
exit 0 and mode 0555. All 14 installations were version-backed.
Independent verification checked all 25 versions: 23 remote objects, two
bundled database snapshots, zero blockers.
- The deployed image is `9200e3fcb4`. Later master integration, the
client-only delayed-reload draft guard and the CLI bootstrap have
separate automated proof. The CLI bootstrap was also exercised against
the tenant's stored configuration and storage through the same read-only
services. This PR does not claim that the deployed image is the final PR
revision.
- Live Studio checks exposed worker wiring, admission, access scope,
exact selection and completion-publication gaps. They are fixed. Failed
attempts remain in the evidence. The full local suite exceeded 80
minutes under host contention and was stopped. Later combined/isolated
local attempts also timed out; they are not counted as passes. Full CI
covers the integrated suites. Earlier CI found an obsolete restart-scope
fixture, which was corrected and passed all 52 HTTP cases.
- Reproduction and migration:
[`doc/skill-ephemeral-web-servers.md`](doc/skill-ephemeral-web-servers.md).
Source revisions, deployments, versions, run IDs, hashes and failure
history:
[`doc/plans/2026-10-30-ephemeral-web-skills.md`](doc/plans/2026-10-30-ephemeral-web-skills.md).

## Risks

- Legacy conversion requires stopped admission and retired
filesystem-authoritative writers. Pre-conversion binaries are not
supported rollback targets afterward.
- Cloud readiness requires remote object storage. Local provider
receipts and provider mismatches remain blockers until objects are
copied and verified.
- Linked local sources retain filesystem semantics and are outside the
cloud guarantee.
- Cold reads still fetch the existing archive format. Warm runtime
preparation retains the verified-cache path.
- Bundled reconciliation preserves an installed head. Existing release
snapshots remain selectable. An older server must not replace a newer
head with its shipped files.
- No production volumes, fleet defaults, or general deployment
coordination change in this PR.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, repository tools, code
execution, and browser use. The exact backend model ID and
context-window size are not exposed in this session. No subagents were
used.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-30 16:18:27 -05:00
DottaPaperclipCodex
 		b06ea111a2 fix(server): schedule skill cache retention from changes and saved deadlines (#17923) 
<!-- entire-trail-link-start -->
https://entire.com/gh/paperclipai/-/trails/83/changes/paperclip/83
<!-- entire-trail-link-end -->

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The server caches saved skill versions on disk so runs can restore
their exact instructions.
> - The old retention owner scans company rows every hour, even after
all caches are protected or removed.
> - Cache publication, version changes, and pin release already have
local or transaction-owned boundaries.
> - Grace windows and terminal controller leases already have saved
timestamps.
> - This pull request uses those changes and deadlines to drive the
existing bounded prune service.
> - Idle retention stops SQL and directory reads while pending cleanup
remains recoverable.

## Linked Issues or Issue Description

Stack slice 64. Parent: #17876 (`codex/sleep-63-agent-timer-deadlines`).
Related: #15625 establishes transaction-owned work signals. #17907
independently keeps an active cache-lock transaction alive; this PR
preserves that lock and does not duplicate that repair.

**What happened?**

At parent `31c12f1d68ff4a51e06fc2be2abf2c2aee9f9d3a`, skill-version
retention has an hourly company scan, a startup timeout, and separate
continuation timers. Empty or permanently protected caches still cause
recurring SQL.

**Expected behavior**

Reconstruct once, inspect changed companies, and schedule saved
access/staging grace or terminal controller deadlines. Stop recurring
work when no timed or uncertain obligation remains. Keep
current/newest/agent/run protection, restorable-byte validation, cache
locking, final pin revalidation, atomic rename, and trash recovery.

**Steps to reproduce**

Start an enabled server with no cached versions, or only current/pinned
versions. After the initial scan, advance an hour. The old interval
selects companies again. The new worker has no deadline and performs no
retention SQL or directory scan in that state.

## What Changed

- Signal the missing-local-source reconciliation deletion in its
transaction. Limit fresh scans of a successfully observed company to one
per thirty seconds. Keep one-second page continuations and actual saved
deadlines independent. Empty or uncertain passes do not delay first
publication.
- Restart a company scan if drain interrupts inspection or post-rename
trash cleanup. Restore bounded candidate/byte logging with dry-run mode.
Three controlled cases failed before repair and the complete 34-case
retention suite passes afterward.
- Register retention through the delivery coordinator and shared
scheduler. Remove the hourly, startup, and continuation timers from
`index.ts`. Track startup work without delaying readiness. Stop delivery
work before closing retained iterators.
- Keep one active company cursor and bounded 50-entry passes. Preserve
observed deadlines across pages and bound company discovery to 51 rows
per page. Retain concrete company failures and incomplete pins for
retry.
- Report saved grace and terminal controller deadlines from the existing
cache owner. PostgreSQL remains the controller-lease authority.
Current/newest/agent pins do not arm unrelated lease checks.
- Wake on cache publication, rejected materialization, and access
markers. Signal version creation/deletion and agent configuration/hire
changes in their owning transactions. Extend existing instruction,
finalization/environment, and retained-session fanouts with company
hints.
- Route reused provider-task prior-owner retirement through
`writeEnvironmentCleanup`, so that supported pin-release write also
settles a notification.
- Add an optional scope callback to `registerDeadlineWork`. Existing
consumers omit it and keep their behavior. The feature owns persistence
predicates; startup only registers it. Add no table or migration.

Full producer audit, shared-contract explanation, lifecycle evidence,
and limitations:
[`doc/plans/2026-10-27-skill-version-retention-sleep-audit.md`](doc/plans/2026-10-27-skill-version-retention-sleep-audit.md).

## Verification

- Reconciled all existing sleep changes with actual master
`7e27bc5007c8f9cb8cdce06c24f9813e27f7120f` through normal merges. All 24
retention-owned and 95 unrelated upstream paths are preserved; package,
UI, E2E and map trees equal master.
- Combined-source qualification: **1,009 TypeScript cases across 42
complete files**, plus **28 Rust cases** in two targets. One intentional
Linux-inotify-only adapter case is skipped on this macOS host. Coverage
includes actual retention/cache/producer/runtime behavior, startup,
shared deadline consumers, changed OpenCode/observability and native
execution boundaries.
- After the subsequent task-index/task-read upstream merges, **85 cases
across six complete files** passed on final source
`d9277b1858a2230bdb726974c90a0c9bc5c5ebe6`: real retention/cache/work,
task reads/query behavior and concurrent migration repair. Counts
overlap earlier coverage. Fresh PostgreSQL 17 schemas apply all 413
journal entries through index 414.
- Database/adapter/Runner builds, production server and Runner
typechecking, module boundaries, target migration order and generated
TypeScript/Rust release-profile consistency pass. Selected source hashes
and temporary fixture imports restore exactly.
- Reproduced and repaired Greptile's own-company due-cleanup P2 with
identical old-worker-failing/new-worker-passing assertions. The
30-second ordinary scan floor remains; already-due cleanup receives its
owed scan and disarms afterward. Earlier 799-case and fresh-blank-plugin
acceptance remain in the audit.
- **Inherited validation exception:** the separate Copilot
source-identity check fails after upstream OpenCode reserved-environment
changes; all 72 identity inputs equal current master. No provider
release attestation is regenerated or new live qualification claimed
here. Required exact-head CI/security and Greptile 5/5 still must pass.
- Historical results, repair details and limitations remain in [the
retention
audit](doc/plans/2026-10-27-skill-version-retention-sleep-audit.md). No
full local repository-suite, live provider, production-load,
external-alarm, measured suspend/wake or fleet-savings claim.

- CI on the prior published head found four connection-review assertions
racing the concealed conversation after reload. Its fresh
Greptile/Ponytail reviews were both 5/5, but the failing browser gate
was not waived or retried unchanged. The fixture now foregrounds the
task tab and waits for visible history before retaining all existing
saved-result assertions. The full file passes four journeys locally; the
controlled-restart case is intentionally skipped by the default
configuration. Original local journeys also passed unchanged, so no
deterministic local red reproduction is claimed. See the audit for CI
failure artifacts and scope. Runtime and prior sleep-test sources remain
unchanged.

## Risks

- Supported live writers are the existing service/API owners in one
process. Independent SQL/FS writers need an ingress contract. Startup
reconstruction covers restarts.
- Notifications cause a bounded inspection; they do not authorize
deletion. Durable pin predicates and the cache lock remain
authoritative.
- A database clock behind application time can require another
one-second check of the concrete expired controller lease. Unknown pins,
lock contention, and failed inspection retain retries until resolved.
- The in-process scheduler cannot wake stopped compute. This PR
preserves Cloud sleep-safety gates and external-alarm coverage. It does
not prove deployed suspension or fleet savings. Cache reclamation can
wait until a supported wake when compute is stopped.
- Native-harness backup retention and the other remaining duties are
separate stack slices.

## Model Used

OpenAI Codex (GPT-6-based coding agent), using repository inspection,
normal Git merges, composition proofs, a controlled red/green regression
and complete PostgreSQL/filesystem integration suites. No subagents were
used in this reconciliation.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Codex <noreply@openai.com>
 2026-10-29 10:14:12 -05:00
Devin FoleyDottaPaperclip
 		86c965ce7a fix(skills): expire unused version caches and restore saved contents (#16433) 
Expire idle skill-version caches with durable runtime pins, bounded discovery, conservative retention, and database-backed restoration.

Co-Authored-By: Dotta <cryppadotta@users.noreply.github.com>
Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-15 12:20:57 -05:00