mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-09 07:10:34 +01:00
DottaandClaude Opus 5.5
b2af45f0f1
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work.
> - New users often connect a Codex agent with an OpenAI API key during
onboarding.
> - A user asked such an agent for a logo. The agent said that the
built-in image generator was not available and that its fallback needed
a separate API key. It then drew an SVG by hand.
> - #19516 removed `--disable image_generation`. That change is correct,
but it does not reach this path. The pinned Codex 0.160.0 shows its
`image_gen` tool only when Codex signs in with ChatGPT. The check is
`image_generation_available` in `codex-rs/core/src/tools/spec_plan.rs`.
An API key never gets the tool.
> - The runner keeps the provider key out of the agent's shell on
purpose, so the skill's `scripts/image_gen.py` fallback cannot run.
> - This pull request adds a Paperclip `generate_image` tool. The server
makes the image with the agent's existing OpenAI key. The key never goes
to the agent process or the shell.
> - The benefit is that API-key Codex agents can make images with no
second key. The spend goes to the agent's budget and the budget can stop
it.
## Linked Issues or Issue Description
Refs #19516
**What happened?**
An agent connected through onboarding with an OpenAI API key (Codex
harness, `gpt-5.6-sol`) could not generate images. The agent said: "The
image skill's built-in generator isn't available in this run, and its
fallback would require a separate API key." It then drew an SVG instead.
**Steps to reproduce**
1. Create a company with a Codex agent that uses an OpenAI API key.
2. Ask the agent for a raster image, for example a logo PNG.
3. The agent reports that image generation is not available. It does not
make the image.
**Expected behavior**
The agent makes the image with the OpenAI key that already powers it.
**Root cause**
The agent ran on the native runner's Codex app-server path, with
`auth.json = {"OPENAI_API_KEY": ...}`. Codex 0.160.0 removes
`image_gen.imagegen` from the model's tools unless
`current_auth_uses_codex_backend()` is true. That function is true only
for a ChatGPT sign-in.
I proved this with the real pinned binary. I gave it the runner's
isolated launch arguments and the API-key auth file, and pointed it at a
local Responses endpoint. The first model request offers only `exec`,
`wait`, `request_user_input` and the collaboration tools. #19516's test
checked only `codex features list`, so it could not see this gate.
## What Changed
- **New `server/src/services/native-runtime/generate-image-tool.ts`:**
- The tool schema and definition.
- `imageGenerationBindingForRun`. It applies only to native Codex runs
on a direct OpenAI `api_key` connection with no router selection.
- The OpenAI Images API call (`gpt-image-2`, `b64_json`, PNG check, 180
s timeout). The reply is read as a stream with a byte cap before it is
parsed. A decoded image larger than 20 MiB is refused.
- A reviewed rate card for `gpt-image-2`: text input $5, image input $8
and image output $30 per 1M tokens, from
developers.openai.com/api/docs/pricing, retrieved 2026-11-06.
- A create-only, symlink-confined local write.
- **`paperclip-runner-tool-authority.ts`:**
- Adds the `imageGeneration` binding.
- Advertises the tool only for standard mode with a workspace. A remote
target must also have an installer. Dot, Muse, Grok and native review do
not get the tool.
- Adds the `#generateImage` handler. Before any spend, it checks again:
the run and task, budget authority, `issue:mutate`, case handoff, and
the connection.
- Stores a claim-before-spend per-call receipt in
`resultJson.imageToolReceipts`. A retry or recovery replays the receipt
and does not pay again.
- Saves a `charged` receipt (usage, price, request ID and charge time)
as soon as OpenAI returns. A failed save, failed accounting write or
crash can then still settle the spend. A retry or recovery records the
cost once and never buys again.
- Records one `cost_events` row through `createCostEventInTransaction`.
The row is `metered_api` with `costStatus: "estimated"`, and the budget
is evaluated. The row is recorded even if saving fails.
- Checks the task's project budget, as well as the company and agent
budgets, before spending.
- Writes content-free `runner.image_generate` activity entries: started,
completed, failed and unconfirmed.
- **`ai-connections.ts`:** adds `selectImageGeneration`. It checks again
that the connection is an active, healthy OpenAI `api_key` connection
with no custom routing. It also checks the responsible user's
membership, the grant audience and the agent or company installation.
These are the same rules that runtime selection uses.
- **`native-session-executor.ts`:** gets the binding from the run's
admitted `contextSnapshot.aiConnection`. The value comes only from the
server, never from tool input.
- **`native-session-resume.ts`:** the native tool contract changes from
`v18` to `v19` and now lists `generate_image`. A persisted provider
thread cannot add tools when it resumes, so older threads must rotate.
- **`doc/runner-api-tools.md`:** a new section about image generation
for OpenAI API-key Codex agents.
- **#19516: kept, not reverted.** Its changes have no effect on the
API-key path. They are still correct for Codex agents that sign in with
ChatGPT, because `image_gen` works there and saves to
`CODEX_HOME/generated_images`. Those agents need the read grant that
#19516 added. On the API-key path, the new tool replaces the built-in
tool. The tool's description tells the agent to use it instead of
`image_gen` or `scripts/image_gen.py`.
## Verification
**Red on `master` (e482f0f632).** Behavioural red with the first version
of the new test:
```
FAIL src/services/native-runtime/generate-image-tool.test.ts > generate_image for OpenAI API-key Codex agents (N-12) > lets an API-key Codex agent make an image with its existing OpenAI key, saved in its workspace and billed
AssertionError: expected [ 'connections_search', …(51) ] to include 'generate_image'
❯ src/services/native-runtime/generate-image-tool.test.ts:78:60
Tests 1 failed | 3 passed (4)
```
The final test file on master's source also fails. The reason is that
the module does not exist: `Cannot find module
'./generate-image-tool.js'`.
**Real pinned Codex binary.** The new test is
`packages/paperclip-runner/src/drivers/codex/codex-api-key-image-tools.test.ts`.
It uses the runner's isolated launch, the API-key `auth.json` and a
local Responses endpoint. It shows that the model gets no `image_gen`,
`imagegen` or `image_generation` tool, and that a server-advertised
`generate_image` tool does reach the model:
```
Test Files 1 passed (1)
Tests 1 passed (1)
```
**Green after the fix:**
`server/src/services/native-runtime/generate-image-tool.test.ts` uses
embedded PostgreSQL and a stubbed OpenAI fetch.
```
Test Files 1 passed (1)
Tests 11 passed (11)
```
The 11 tests cover these behaviours:
- **Main path.** The tool is advertised. The request goes to
`api.openai.com/v1/images/generations` with the existing key. The PNG is
written to the workspace. The key does not appear in the result or in
activity. One `cost_events` row is written: 50 text tokens and 1000
output tokens, which is $0.03025, estimated, from the rate card. A retry
and `recover()` replay the receipt with no second fetch and no second
cost row.
- **No binding:** the tool is not offered and `execute` refuses.
- **Budget hard-stop.** The stop pauses the agent, so the call is
refused before any fetch and no file is written.
- **Bad grant:** the call is refused before any fetch.
- **Project budget stop.** The task's project is paused by its budget,
so the call is refused before any fetch.
- **Charge settlement.** A crash left a `charged` receipt. The retry and
`recover()` record the cost once, report the image path and do not buy
another image.
- **Oversized reply.** The read stops before the body is parsed.
- **Save failure.** The spend is still recorded. A same-call retry does
not pay again and reports the failure.
- **Binding gate.** Only Codex on a direct OpenAI API key gets the
binding. A subscription, ACPX, OpenRouter or a routed connection does
not.
- **Rate card math.**
- **Provider errors.** The error text does not contain the key, and
non-PNG data is rejected.
**Neighbouring suites:**
- `src/drivers/codex` in the runner package: 24 files, 350 tests passed.
- Server: `paperclip-runner-tool-authority.test.ts`,
`native-session-resume.test.ts`, `fast-responses.test.ts`,
`ai-connection-session.test.ts` and the new suite. 157 of 158 passed.
- The one failure is `native-session-resume.test.ts › archives a damaged
prior epoch … with real runnerd` (ENOENT on `authority-epochs`). It
fails the same way on unmodified `master` in this local environment, so
this change did not cause it.
- `tsc --noEmit -p server/tsconfig.json`: clean.
- `pnpm verify` exits 2 (broad). Required CI supplies the broad
coverage.
## Risks
- **New security surface: this needs maintainer sign-off.** The server
now sends the agent's OpenAI API key to `api.openai.com` on the agent's
request. The tool is limited in these ways:
- The key stays in the server.
- Each call checks run, task, budget, mutation and connection authority
again.
- The destination name comes from the server and must match
`^[a-z0-9-]+-[0-9a-f]{8}\.png$`. The write is create-only and confined
to the workspace root. On macOS it uses `O_NOFOLLOW_ANY`. On Linux it
uses a `/proc` descriptor walk.
- **Spend.** Each call costs money. The tool description tells the agent
to make only what the task needs. The cost is an estimate from a rate
card (`costStatus: "estimated"`), not an invoice. The rate card applies
the image-output rate to all output tokens, so the estimate errs on the
high side.
- **Checkpoint rotation.** The tool-contract bump from v18 to v19 makes
existing native provider threads start fresh once at their next turn.
This is what the contract requires when the tool surface changes.
- **Pricing drift.** If OpenAI changes `gpt-image-2` prices, the rate
card must be updated.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, sub-agents) and extended
reasoning.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>