mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-22 11:54:35 +01:00
a792409d07e984e7bf8382e295bdaebd2aa18dda
3
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
e2cee049f9 |
fix: show instruction saves in history, credit the real writer, and speed up the agent tool-rule list (#19515)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Operators and agents change agent instructions (AGENTS.md), and they
must see who changed them and when
> - Agent files replaced instruction revisions. Saves no longer write
revision rows, so instruction history is empty
> - The current revision is stamped with the reader and the date
1970-01-01, so it gives false audit data
> - Separately, an agent that lists tool rules gets a timeout when many
rules name the same resources
> - This pull request reads history from the save records that Paperclip
already writes, credits the real writer, and checks each tool-rule
resource one time
> - The benefit is a correct audit trail for instruction changes and a
tool-rule list that agents can read
## Linked Issues or Issue Description
No public issue exists. I searched open and closed issues and PRs for
these problems and found none.
**What happened?**
1. `get_agent_instruction_history` (and the history API) returns 0
revisions for an agent whose AGENTS.md was saved many times. The
activity log shows each save as `agent.files_updated` with the path,
content hash, actor and run.
2. `read_agent_instructions` returns a current revision with `createdAt`
1970-01-01. Its actor is the agent that reads it, and its `sourceRunId`
is the run that reads it.
3. `GET /api/companies/{companyId}/tools/policies` from an agent fails
with a transport timeout most of the time when the company has many tool
rules. A board user gets the list without a problem.
**Expected behavior**
1. History lists each save, newest first, with its writer, run, source
and time.
2. The current revision names the agent or user that wrote those bytes,
and the time of that save. If no save is recorded, it names no writer.
3. The agent gets the tool-rule list in normal time, with the same rules
hidden as before.
**Steps to reproduce**
1. Save an agent's instruction entry file (board editor,
`update_agent_instructions`, or a run that edits the file). Read the
history. It is empty.
2. As a different agent, call `read_agent_instructions` on that agent.
The revision shows the reader and 1970-01-01.
3. Create about 40 tool rules that name the same connection and agent.
As an agent, list `/tools/policies`. Each rule does a new set of access
checks for each resource, one after another.
**Paperclip version or commit**
`master` at `819420a60a`.
**Deployment mode**
Hosted. The new tests reproduce all 3 problems on `master`.
## What Changed
- `agent-instruction-revisions.ts`: history merges the legacy revision
rows with the `agent.files_updated` save records for the entry file. The
order is `(created_at, id)` descending, and cursors keep PostgreSQL
timestamp precision.
- `agent-instruction-revisions.ts`: the current revision takes its
writer, run, source and time from the latest save of the same content
hash. A reader is never the writer. Bytes with no save record have no
writer.
- `agent-instruction-revisions.ts`: an identical re-save over bytes with
no record is marked `source: "adopted"`. It repairs the runtime journal
but is not listed as a save.
- `agent-instruction-revisions.ts`: a history ID can be a legacy
revision, a save record, or the current content token. A save record
keeps only metadata. If its bytes are no longer current, a read returns
404 with `INSTRUCTION_REVISION_CONTENT_NOT_RETAINED`.
- `agent-file-store.ts`: saves record `byteLength`. A run collection
that changes the entry file records its path, hash and size, so history
shows saves from runs.
- `connection-management-authz.ts`: `filterConnectionManagementPolicies`
keeps one read decision for each distinct resource in a request, and
loads each application or catalog entry one time. A rule is still hidden
when any resource that it names is hidden. Board callers do not use this
path.
- Tests: new tests for save history, paging, writer credit, adoption,
and the tool-rule list. Old assertions that expected empty history now
check the exact saves.
## Verification
- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/agent-instruction-revisions.test.ts
src/__tests__/agent-instruction-working-copies.test.ts
src/__tests__/agent-directory-working-copies.test.ts
src/__tests__/built-in-agents.test.ts
src/services/native-runtime/agent-instruction-tools.integration.test.ts
src/__tests__/tool-policy-list-authz.integration.test.ts`
- With the fix: all pass (revisions: 28 passed, 1 skipped on Linux;
other 5 files: 230 passed).
- With the new tests on `master` code: 13 tests fail, as expected.
- `src/__tests__/connection-management.integration.test.ts`: 90 passed.
- The new tool-rule test lists 40 rules that name 1 connection and
counts the connection reads. It needs fewer than 10.
- I could not run the full server typecheck in my sandbox because of a
memory limit. CI runs it.
- CI on head `a2f0b825`: all checks pass, including `ci / verify` and
`ci / e2e`.
## Risks
- History now shows entries whose ID is an activity row ID, not a
revision row ID. Clients that read or restore an older save get a clear
404 because Paperclip no longer keeps those bytes. The latest save and
legacy revisions still read and restore as before.
- Saves recorded before this change have no `byteLength`. For those,
history shows the current size when the hash matches, and 0 otherwise.
- Stock updates of built-in agents do not record a file path, so they do
not show in history.
- No migration. No change to the stored data model.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), in Claude Code with
tool use (shell, file edits, test runs). Standard context window.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: lucktastic <13757243+lucktastic@users.noreply.github.com>
|
||
|
|
b9cf64f5bf |
test(server): check the agent row lock with FOR SHARE so lock checks cannot block each other (#21775)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Each master commit must pass Cloud readiness before its canary and image publish > - `agent-file-restore-locking.test.ts`, added in #21694, failed master Cloud readiness run 43712563988 on 2026-11-15, so that commit's canary did not publish > - The test reported the agent row as locked during the download, but its own sampler measured 0 ms of product lock time > - The cause is in the test: its two lock checks used `FOR UPDATE NOWAIT` on one pool and could block each other > - This pull request checks the lock with `FOR SHARE NOWAIT`, which still detects the product's `FOR UPDATE` but never conflicts with another check > - The benefit is that the test fails only when the product really holds the lock ## Linked Issues or Issue Description **What happened?** `downloads a stale projection without holding the agent row lock` failed in master Cloud readiness run 43712563988, attempt 2: `expected [{ objectKey, agentRowLocked: true }] to deeply equal [{ objectKey, agentRowLocked: false }]`. The test's metric line showed `lockedMs: 0`. **Steps to reproduce** 1. Make each sampler check hold its `FOR UPDATE NOWAIT` lock for 50 ms (to make the overlap certain). 2. Run `pnpm exec vitest run src/__tests__/agent-file-restore-locking.test.ts -t "stale projection"` in `server/`. 3. The test fails 3 of 3 times with `agentRowLocked: true` and `lockedMs: 0`, the same as CI. **Expected behavior** Only a lock held by the product makes the check report "locked". Refs #21694 ## What Changed - `agentRowLocked()` uses `FOR SHARE NOWAIT` instead of `FOR UPDATE NOWAIT`. The product takes `FOR UPDATE` on this row (`agent-file-store.ts`, `agent-instruction-revisions.ts`), which conflicts with `FOR SHARE`. Two `FOR SHARE` checks do not conflict. - New test: three concurrent checks all report unlocked, a `FOR UPDATE` held on another connection is reported locked, and the row reads unlocked again after release. ## Verification - Red: with the forced 50 ms overlap and the old `FOR UPDATE` probe, the test failed 3 of 3 runs with the CI error. - Green: with the same forced overlap and `FOR SHARE`, 4 of 4 tests passed in 3 of 3 runs. - Final file (no forced overlap): `pnpm exec vitest run src/__tests__/agent-file-restore-locking.test.ts` in `server/`: 5 of 5 tests passed in 5 of 5 runs. ## Risks - Low risk. Test-only change. The new test makes sure the check still detects a real `FOR UPDATE` lock. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [ ] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
|
c38a471042 |
perf(server): download stale agent files before the agent row lock (#21694)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Durable agent files live in object storage. Each server keeps a local copy (the projection), marked with the head generation it matches > - When the projection is stale, the next locked read or write restores it: it downloads every file from object storage. This happens for the first run of each agent on a new server (every deploy on hosted ephemeral servers) and after another server wrote > - The download ran inside the transaction that holds the `agents` row lock (`FOR UPDATE`), on the main pool for the run-start read (`readCommittedForRuntime`) and on the 2-connection file-storage pool for the store. A download can take up to 30 s > - After a deploy at 1,000 agents, this is one long row lock and one pinned connection per agent, all at once, and every writer of that agent waits behind it > - This pull request downloads the files outside the transaction and takes the lock again only to swap them in > - The benefit is that the row lock is held for milliseconds instead of the download time (0 ms observed instead of 321 ms with a 300 ms download in the test), and writers do not wait for a download ## Linked Issues or Issue Description **What existing behavior does this improve?** The restore of a stale local copy of an agent's durable files (the first run of each agent on a new server, or after another server wrote). **Subsystem affected** Server: durable agent files (`agent-file-durability.ts`, `agent-file-store.ts`, `agent-instruction-revisions.ts`). **Current behavior** The restore downloads every file from object storage inside the transaction that holds the `agents` row lock (`FOR UPDATE`) and a pooled connection. A download can take up to 30 s, and every writer of that agent waits behind it. **Proposed behavior** Under the lock, detect that a download is needed and roll back. Download the head's files, verified by digest, into a private staging directory with no lock. Run the transaction again, and swap the staged files in only when the head generation is unchanged. After two changed heads, restore under the lock as before. **Reason and benefit** After a deploy at 1,000 agents, every agent's first run restores at once. The row lock and connection are now held for milliseconds instead of the download time, and writers do not wait for a download. **Additional context** Refs the 1,000-concurrent-agents plan (`doc/plans/2026-11-13-thousand-concurrent-agents.md`). The save path had the same problem for uploads and was fixed earlier with uploads before the lock (`uploadAgentFilesBeforeCommit`); this is the restore side. Reading the head before every transaction was considered; it adds a query to every call, while the deferral costs nothing when the projection is current. ## What Changed - `agent-file-durability.ts`: the download loop moves into `downloadAgentFiles`, unchanged. `withAgentFilesRestoredBeforeLock(work)` runs `work` in a scope. Inside the scope, the first restore of an attempt that needs a download throws a private deferral error, which rolls back the transaction. The wrapper stages the download in `.agent-files-<uuid>` next to the root and runs `work` again. `restoreDurableAgentFiles` uses a staged directory only when its generation equals the head it reads under the lock. Staged directories are removed when the scope ends. - Only the first restore of each attempt can defer, so nested restores of other agents keep their old behavior. Outside the scope, the behavior is unchanged. - `agent-file-store.ts` (`locked`, so every store read, write, apply and working-copy preparation) and `agent-instruction-revisions.ts` (`readCommittedForRuntime`, `readCurrent`, `materializeCurrent`) run their transactions in the scope. ## Verification - New test `server/src/__tests__/agent-file-restore-locking.test.ts` on embedded PostgreSQL, with an object-storage fixture: - Hold time, with a 300 ms download, through `readCommittedForRuntime` (the run-start read). Before: the download ran with the agent row locked, and the row was locked 321 ms of 322 ms. After: the download ran with the row unlocked, and a sampler saw the row locked for 0 ms. - Concurrency: a board write of the same file runs while the reader's download is blocked. Before: the writer waited for the reader's download (5 s timeout). After: the writer commits a new head during the download. The reader then discards its stale staged copy and returns the writer's content. The projection is marked at the writer's generation, and no staging directory is left. - Failure: when object storage fails, the read fails once with the storage error, after one attempt made without the row lock, and leaves no staging directory. - Progress: when the head generation changes during each of two downloads, the third attempt restores under the lock as before (3 downloads), and the projection matches the final head. - `npx vitest run` for agent-file-checkpoint-locking, agent-file-checkpoints, agent-file-durability, agent-file-restore-locking, agent-instruction-revisions, agent-instruction-working-copies, agent-instructions-service, agent-directory-working-copies and agent-instructions-routes: 287 passed, 1 skipped. - `npx tsc --noEmit -p server/tsconfig.json`: passes. ## Risks - A deferred attempt runs the transaction body again. The deferral happens at the first restore, before the body writes files, and the transaction rollback undoes database work, so the second attempt starts from the same state. - A staged download uses disk next to the agent root until the scope ends, the same as the temporary directory of the old download. - If the head changes during every download, the third attempt downloads under the lock as before. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use and code execution. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> |