mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 21:34:48 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - Native agent runs read the Paperclip API through `call_api`. Large responses are saved as "API snapshot" assets so the agent can read them in pages. > - Each company has a 20 GiB snapshot quota. Before a large read, the server reserves 1 GiB of worst-case space against that quota. > - Snapshot assets were never removed. Reservations from interrupted captures (crash, restart, ambiguous upload) stayed at 1 GiB each until "an operator reconciles" them. Nothing did that. > - After a morning of high retry volume, the quota filled. Every large read then failed with `api_response_company_storage_limit`, and agents blocked their own tasks and asked an operator to free storage. > - This pull request makes company admission reclaim space by itself: it releases stale reservations of finished runs and evicts the oldest snapshots of finished runs, oldest first, only as much as the new read needs. > - The benefit is that a read never fails because old snapshots filled the quota, and no person must clear storage by hand. ## Linked Issues or Issue Description **What happened?** On staging, about 97 tasks of one agent had successful runs but ended in `blocked`. In each run, `GET` calls through `call_api` for large records (for example heartbeat runs and their events) failed with `api_response_company_storage_limit`: "Company API snapshot storage quota is exhausted. Read an existing asset or ask the operator to remove old snapshots or increase the quota." The agent then reported a task-wide blocker owned by "the instance storage operator". At least 15 tasks had a final comment such as "Blocked: company API snapshot storage is exhausted". The error message itself told the agent that an operator must act. **Expected behavior** A read must not fail because old snapshots filled the company quota. The server must reclaim the space automatically. Only snapshots that active runs still hold, or that users attached to issues, cases or logos, may stay. **Steps to reproduce** 1. Set `PAPERCLIP_RUNNER_API_COMPANY_CAPTURE_MAX_BYTES` to 2 GiB. 2. Store a 1 GiB `runner-api` snapshot asset from a finished run, or leave two 1 GiB unattached reservations from finished runs that are older than 20 minutes. 3. In a running native run, call `call_api` for a response larger than the inline limit. 4. Before this change, the call fails with `api_response_company_storage_limit`. After this change, the call succeeds and the oldest finished snapshot is evicted. **Paperclip version or commit** Staging deploy `704b3f4d63`. Fix is based on `origin/master` at `8bdd53c11f`. **Deployment mode** Hosted (staging cloud), native runtime with Daytona sandboxes. **Agent adapter(s) involved** Native Paperclip runner (Codex app server, `gpt-6.1-sol`). The defect is in the server-side `call_api` capture admission, so it affects every native adapter. ## What Changed - Added `server/src/services/native-runtime/runner-api-capture-retention.ts`. It measures company snapshot usage and reclaims space under the existing per-company advisory lock. - Reclaim step 1: release unattached reservations older than twice the capture deadline (20 minutes) whose run is missing or not `queued`, `running` or `scheduled_retry`. - Reclaim step 2: evict snapshot assets oldest first until the new reservation fits. A snapshot is eligible only when it is older than 1 hour, every run that captured it has finished (succeeded, interrupted, failed, cancelled or timed out), and no issue attachment, case attachment or company logo links to it. Each eviction writes an `asset.evicted` activity-log row with the asset id and the owning run id. - A capture upload that finishes after its reservation was reclaimed is rejected in the publish transaction. The same transaction records the object for deletion, and the bounded cleanup removes it. The upload has a 10-minute abort signal. - Failed deletes are logged per company and counted with the `runner_api_capture_delete_failed` metric. When cleanup keeps failing, the read error says so in plain words. - Eviction credits deletes that are already pending before it chooses more snapshots. Eviction is durable. In the same transaction that deletes an asset row, it inserts a reservation row with `pending_delete_object_key`, so the bytes stay counted. Admission then deletes the stored objects in pages, under a 30-second abort signal, and removes each row only after its delete succeeds. A failed or interrupted delete is retried at the next over-quota admission. No background poller is added. - Added the `pending_delete_object_key` column and an index on `runner_api_response_reservations.asset_id` (migration 0472). - `#reserveApiResponseCapture` now runs reclaim only when the quota would be exceeded, so reads below the quota do no extra work. After marking evictions it finishes the deletes and retries admission once. - When no space can be reclaimed, the error message now names what holds the space (runs still working, files attached to tasks, deletes in progress). It no longer tells the agent to ask an operator. - Replaced the test that asserted old snapshots block reads with tests for eviction, multi-page eviction, durable delete retry, a bounded stalled delete, stale-reservation release, and protection of active and attached snapshots. ## Verification - Red: the three new tests in `runner-api.integration.test.ts` fail on `origin/master` (two get `api_response_company_storage_limit`, one gets the old "ask the operator" message). - Green: `cd server && npx vitest run src/services/native-runtime/runner-api.integration.test.ts` passes 70/70 with the fix. - The multi-page test failed first with `api_response_company_storage_limit` because pending deletes ran in one 200-row batch. The paged delete loop fixed it. - `cd server && npx tsc --noEmit -p .` passes. ## Risks - An agent that reads a saved snapshot of a finished run after it was evicted gets a not-found response. It can read the API again. Eviction happens only under quota pressure, oldest first. - A released stale reservation can leave an unknown storage object from an ambiguous upload. These bytes were never linked to an asset, so they were already unreachable. This is the same storage the old code kept counted forever. - Eviction runs inside the company capture lock. The candidate query is bounded to pages of 200 rows and uses the new `asset_id` index. - Migration 0472 adds one nullable column (`ADD COLUMN IF NOT EXISTS`) and builds one index with `CREATE INDEX CONCURRENTLY IF NOT EXISTS`. It is listed in `MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS`. It is additive and safe for old and new servers during a rolling deploy. - During a storage outage, an over-quota read still fails until deletes succeed. The bytes stay counted, so the quota is never exceeded. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude Code with tool use (shell, file edits, browser reads of staging). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
137 lines
8.3 KiB
TypeScript
137 lines
8.3 KiB
TypeScript
import { and, asc, eq, gt, inArray, isNotNull, isNull, lt, sql } from "drizzle-orm";
|
|
import { activityLog, assets, runnerApiResponseReservations, type Db } from "@paperclipai/db";
|
|
import type { StorageService } from "../../storage/types.js";
|
|
import { logger } from "../../middleware/logger.js";
|
|
import { incrementToolRuntimeMetricCounter } from "../tool-runtime-metrics.js";
|
|
import { RUNNER_API_RESPONSE_DEADLINE_MS } from "./runner-api-response-limits.js";
|
|
|
|
const ACTIVE_RUN_STATUSES = ["queued", "running", "scheduled_retry"] as const;
|
|
const TERMINAL_RUN_STATUSES = ["succeeded", "interrupted", "failed", "cancelled", "timed_out"] as const;
|
|
export const CAPTURE_EVICTION_MIN_AGE_MS = 60 * 60_000;
|
|
export const RUNNER_API_CAPTURE_EVICTION_PASSES = 2;
|
|
export const RUNNER_API_CAPTURE_RETENTION_ACTOR = "api-response-retention";
|
|
const EVICTION_PAGE_SIZE = 200;
|
|
const PENDING_DELETE_BATCH_SIZE = 50;
|
|
export const STALE_CAPTURE_RESERVATION_MS = 2 * RUNNER_API_RESPONSE_DEADLINE_MS;
|
|
export const CAPTURE_EVICTION_DELETE_TIMEOUT_MS = 30_000;
|
|
export const CAPTURE_CLEANUP_STALLED_MS = 5 * 60_000;
|
|
export const RUNNER_API_CAPTURE_DELETE_FAILURE_METRIC = "runner_api_capture_delete_failed";
|
|
|
|
function activeRunStatusList() {
|
|
return sql.join(ACTIVE_RUN_STATUSES.map(status => sql`${status}`), sql`, `);
|
|
}
|
|
|
|
function terminalRunStatusList() {
|
|
return sql.join(TERMINAL_RUN_STATUSES.map(status => sql`${status}`), sql`, `);
|
|
}
|
|
|
|
export async function runnerApiCaptureUsageBytes(tx: Db, companyId: string): Promise<number> {
|
|
const [row] = await tx.execute<{ bytes: string | number }>(sql`select
|
|
coalesce((select sum(${assets.byteSize}) from ${assets}
|
|
where ${assets.companyId} = ${companyId}
|
|
and (${assets.objectKey} like ${`${companyId}/runner-api/%`}
|
|
or exists (select 1 from ${runnerApiResponseReservations} capture
|
|
where capture.asset_id = ${assets.id} and capture.company_id = ${companyId}))), 0)
|
|
+ coalesce((select sum(${runnerApiResponseReservations.reservedBytes}) from ${runnerApiResponseReservations}
|
|
where ${runnerApiResponseReservations.companyId} = ${companyId}
|
|
and ${runnerApiResponseReservations.assetId} is null), 0) as bytes`);
|
|
return Number(row?.bytes ?? 0);
|
|
}
|
|
|
|
export async function markRunnerApiCaptureEvictions(tx: Db, companyId: string, neededBytes: number, now = new Date()) {
|
|
const released = await tx.delete(runnerApiResponseReservations).where(and(
|
|
eq(runnerApiResponseReservations.companyId, companyId),
|
|
isNull(runnerApiResponseReservations.assetId),
|
|
isNull(runnerApiResponseReservations.pendingDeleteObjectKey),
|
|
lt(runnerApiResponseReservations.createdAt, new Date(now.getTime() - STALE_CAPTURE_RESERVATION_MS)),
|
|
sql`not exists (select 1 from heartbeat_runs run where run.id = ${runnerApiResponseReservations.runId}
|
|
and run.status in (${activeRunStatusList()}))`,
|
|
)).returning({ bytes: runnerApiResponseReservations.reservedBytes });
|
|
let reclaimedBytes = released.reduce((sum, row) => sum + row.bytes, 0);
|
|
const [pending] = await tx.select({ bytes: sql<string>`coalesce(sum(${runnerApiResponseReservations.reservedBytes}), 0)` })
|
|
.from(runnerApiResponseReservations).where(and(
|
|
eq(runnerApiResponseReservations.companyId, companyId), isNotNull(runnerApiResponseReservations.pendingDeleteObjectKey),
|
|
));
|
|
let pendingBytes = Number(pending?.bytes ?? 0);
|
|
while (reclaimedBytes + pendingBytes < neededBytes) {
|
|
const candidates = await tx.select({ id: assets.id, objectKey: assets.objectKey, byteSize: assets.byteSize,
|
|
runId: sql<string | null>`(select capture.run_id from runner_api_response_reservations capture
|
|
where capture.asset_id = "assets"."id" and capture.run_id is not null limit 1)` }).from(assets).where(and(
|
|
eq(assets.companyId, companyId),
|
|
lt(assets.createdAt, new Date(now.getTime() - CAPTURE_EVICTION_MIN_AGE_MS)),
|
|
sql`(${assets.objectKey} like ${`${companyId}/runner-api/%`}
|
|
or exists (select 1 from runner_api_response_reservations capture
|
|
where capture.asset_id = ${assets.id} and capture.company_id = ${companyId}))`,
|
|
sql`not exists (select 1 from runner_api_response_reservations capture
|
|
join heartbeat_runs run on run.id = capture.run_id
|
|
where capture.asset_id = ${assets.id} and run.status not in (${terminalRunStatusList()}))`,
|
|
sql`not exists (select 1 from issue_attachments link where link.asset_id = ${assets.id})`,
|
|
sql`not exists (select 1 from case_attachments link where link.asset_id = ${assets.id})`,
|
|
sql`not exists (select 1 from company_logos link where link.asset_id = ${assets.id})`,
|
|
)).orderBy(asc(assets.createdAt), asc(assets.id)).limit(EVICTION_PAGE_SIZE);
|
|
const chosen: typeof candidates = [];
|
|
for (const candidate of candidates) {
|
|
if (reclaimedBytes + pendingBytes >= neededBytes) break;
|
|
chosen.push(candidate);
|
|
pendingBytes += candidate.byteSize;
|
|
}
|
|
if (chosen.length) {
|
|
await tx.delete(assets).where(and(eq(assets.companyId, companyId), inArray(assets.id, chosen.map(candidate => candidate.id))));
|
|
await tx.insert(runnerApiResponseReservations).values(chosen.map(candidate => ({
|
|
companyId, runId: null, assetId: null, reservedBytes: candidate.byteSize, pendingDeleteObjectKey: candidate.objectKey,
|
|
})));
|
|
await tx.insert(activityLog).values(chosen.map(candidate => ({
|
|
companyId, actorType: "system", actorId: RUNNER_API_CAPTURE_RETENTION_ACTOR, action: "asset.evicted",
|
|
entityType: "asset", entityId: candidate.id, runId: candidate.runId,
|
|
details: { source: "runner.call_api", reason: "company_capture_storage_full", byteSize: candidate.byteSize, runId: candidate.runId },
|
|
})));
|
|
}
|
|
if (candidates.length < EVICTION_PAGE_SIZE) break;
|
|
}
|
|
return { releasedBytes: reclaimedBytes, pendingBytes };
|
|
}
|
|
|
|
export async function finishPendingRunnerApiCaptureDeletes(db: Db, storage: StorageService, companyId: string,
|
|
timeoutMs = CAPTURE_EVICTION_DELETE_TIMEOUT_MS) {
|
|
const signal = AbortSignal.timeout(timeoutMs);
|
|
let deletedCount = 0;
|
|
let failedCount = 0;
|
|
let after: string | undefined;
|
|
while (!signal.aborted) {
|
|
const pending = await db.select({ id: runnerApiResponseReservations.id, objectKey: runnerApiResponseReservations.pendingDeleteObjectKey })
|
|
.from(runnerApiResponseReservations).where(and(
|
|
eq(runnerApiResponseReservations.companyId, companyId), isNotNull(runnerApiResponseReservations.pendingDeleteObjectKey),
|
|
after ? gt(runnerApiResponseReservations.id, after) : undefined,
|
|
)).orderBy(asc(runnerApiResponseReservations.id)).limit(PENDING_DELETE_BATCH_SIZE);
|
|
if (!pending.length) break;
|
|
after = pending[pending.length - 1].id;
|
|
const results = await Promise.all(pending.map(row => storage.deleteObject(companyId, row.objectKey!, { signal, allVersions: true })
|
|
.then(() => ({ id: row.id, error: null as unknown }), (error: unknown) => ({ id: row.id, error }))));
|
|
const deleted = results.filter(result => result.error === null).map(result => result.id);
|
|
const failures = results.filter(result => result.error !== null);
|
|
if (failures.length) {
|
|
failedCount += failures.length;
|
|
logger.warn({ companyId, failedDeletes: failures.length, err: failures[0].error },
|
|
"Failed to delete evicted API response snapshots; they stay counted and will be retried");
|
|
await Promise.all(failures.map(() => incrementToolRuntimeMetricCounter(db, { companyId, metric: RUNNER_API_CAPTURE_DELETE_FAILURE_METRIC })
|
|
.catch(() => undefined)));
|
|
}
|
|
if (deleted.length) {
|
|
await db.delete(runnerApiResponseReservations).where(and(
|
|
eq(runnerApiResponseReservations.companyId, companyId), inArray(runnerApiResponseReservations.id, deleted),
|
|
));
|
|
deletedCount += deleted.length;
|
|
}
|
|
if (pending.length < PENDING_DELETE_BATCH_SIZE) break;
|
|
}
|
|
return { deleted: deletedCount, failed: failedCount };
|
|
}
|
|
|
|
export async function runnerApiCaptureCleanupStalled(db: Db, companyId: string, now = new Date()) {
|
|
const [row] = await db.select({ id: runnerApiResponseReservations.id }).from(runnerApiResponseReservations).where(and(
|
|
eq(runnerApiResponseReservations.companyId, companyId), isNotNull(runnerApiResponseReservations.pendingDeleteObjectKey),
|
|
lt(runnerApiResponseReservations.createdAt, new Date(now.getTime() - CAPTURE_CLEANUP_STALLED_MS)),
|
|
)).limit(1);
|
|
return Boolean(row);
|
|
}
|