Commit Graph
3 Commits
Author SHA1 Message Date
Devin FoleyandPaperclip 3367b75ccc fix: fence accepted work and cleanup before idle sleep (#15522)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A host can stop an idle instance to reduce unused compute.
> - Zero active runs do not prove that requests, saved work, or cleanup
are complete.
> - A client can disconnect while its request still writes, and failed
cleanup can remain only in memory or on disk.
> - This pull request holds new admission and checks accepted work,
cleanup, and persisted work under an owned drain.
> - The host gets an empty report only while those checks remain valid.

## Linked Issues or Issue Description

Refs #13413. That companion PR uses the same owned-hold vocabulary for
runtime services. This PR covers HTTP requests, scheduler work,
accounting, cleanup, and the instance work inventory. It does not
include the preview gateway changes.

**What existing behavior does this improve?**

The instance task-drain API reports process counters. It does not prove
that a host can safely stop the instance for idle sleep.

**Current behavior**

A quiet run set can coexist with an unfinished request, cleanup after a
completed run, future work, or a failed accounting write.

**Proposed behavior**

Provide a bounded owned idle hold. Block new ingress, track accepted
handler promises, inspect durable and local work, and return `none` only
when the same hold stays quiet through the checks. Keep normal
deployment drains compatible.

**Reason and benefit**

Hosts can identify eligible idle instances without treating a disconnect
or failed cleanup write as completed work.

## What Changed

- Add `purpose: "idle"`, a bounded TTL, unique owners, and owner-checked
release to task drain. Existing holds cannot be replaced by another API
request.
- Gate HTTP ingress before parsers, auth, webhooks, and MCP. Gate new
WebSocket upgrades. Track async handlers in nested Express routers and
error middleware until they settle, even after the response or client
disconnect.
- Count accepted live-event WebSocket authentication through settlement,
even after disconnects. Count detached built-in agent, managed-home and
runtime-service startup reconciliation after readiness.
- Keep health and control mutations tracked. Count control-request
authentication separately from the read-only report, including
concurrent user/company/membership writes.
- Pause new scheduler admissions during idle holds. Count work already
in flight, including database backups, and reject scans whose work
generation changes. Periodic backups block sleep without a host wake
schedule.
- Inspect accounting and orphan-cleanup spool directories without
skipping temporary or malformed entries. Retain orphan tokens through
queue splices, flush failures, and buffer overflow. Keep failed usage
capture counted until its database failure fence is written.
- Check persisted work across companies in a bounded read-only
transaction. Include deferred agent-file cleanup and saved watchdogs
whose watched issues are complete. Enabled plugins and unsupported
retained work remain blockers.
- Require the exact idle owner and completed startup tracking before
returning an empty report. Keep reports free of tenant details.
- Document the hosting protocol, retry behavior, conservative blockers,
and the remaining external provider-stop race.

## Verification

- Current head: `8d9a599b732c2047b8c671d4799065d7c90a3567`, rebased on
master `941a3fa991aeb97eb1ac390c65b7973b5f6de1ad`. Both heartbeat helper
extractions are preserved. GitHub confirms no merge conflicts.
- Focused heartbeat renderer/run-log, drain, control-auth, admission,
route and PostgreSQL inventory checks: 234 tests passed in 10 suites
after the rebase.
- The POST task-drain contract includes the expected `409` conflict
response. The 84 OpenAPI and instance-settings route tests and server
typecheck passed after that final documentation fix.
- Final accepted-upgrade/startup regression run: 104 tests passed in
five suites, including success and failure after readiness or
disconnect. Final server typecheck also passed.
- `pnpm -r typecheck`: passed.
- `pnpm build`: passed.
- `pnpm install --frozen-lockfile` and the PR diff secret scan passed.
No dependency or lockfile changes are added by this PR.
- The prior verification covered HTTP disconnects and early responses,
async error handlers, concurrent authentication writes, signed
bootstrap, saved watchdogs, backup promises, orphan cleanup, spools and
failed accounting fences. Those tests passed.
- The last full local `pnpm test:run` stopped in the general-server
phase with 16,658 passing tests and five skill/connector fixture
failures caused by an ancestor workspace skill directory. That full
local run preceded this rebase and has not been repeated for the import
conflict. Full current-head CI passed: 53 successful checks and two
conditional skips, with no failures.
- All eight review findings are fixed and their threads resolved,
including accepted upgrade authentication, detached startup writes and
the POST conflict contract. Current-head Apex review is 5/5 with no new
findings; all eight review threads remain resolved.
- No live provider stop or production change was performed.

## Risks

- The hosting controller must use the owned protocol and hold external
admission through its final validation and provider stop. A legacy drain
cannot authorize idle sleep. During an idle hold, new requests receive
503 with `Retry-After: 1`; the host must handle queueing or retry before
enabling this path.
- This is a single-process protocol. An unexpected restart after the
last validation can race an external provider stop. The host must
serialize deploy/wake/sleep operations and bind the validation to the
instance it stops. Multiple replicas need shared fencing.
- Some retained state conservatively prevents sleep, including every
enabled plugin. This PR does not promise that every inactive instance
becomes eligible.
- The HTTP adapter uses Express 5 router layers. Real Express tests
cover nested routes, errors and disconnects. New routes must register
before tracking is installed. Detached work must have durable state or
explicit work tracking.
- Unrecoverable in-memory cleanup debt keeps the instance awake until
reconciliation. This change does not make such debt survive an unplanned
process crash. No schema migration or provider configuration change is
included.

## Model Used

OpenAI GPT-6 via Codex, with reasoning, repository inspection, tool use
and code execution. The exact deployment variant/model ID and context
window are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run focused tests locally and they pass; the full local
fixture limitation and clean full CI result are documented above
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-08 08:35:45 -07:00
Devin FoleyandPaperclip 202c2d307e fix(server): leave unclaimed warm Cloud databases idle (#15314)
## Thinking Path

> - Paperclip manages work performed by AI agents.
> - Managed deployments prepare empty applications before an owner
claims them.
> - Those applications start database pollers even though no company can
have work.
> - Health probes also query SQL, so idle databases cannot remain
suspended.
> - This pull request adds an explicit standby marker for empty,
unclaimed Cloud apps.
> - The existing signed, durable claim resumes normal processing without
restarting the app.

## Linked Issues or Issue Description

**What happened?**

An empty, unclaimed warm application runs recurring chat, email, plugin,
heartbeat, cleanup, and reconciliation queries. Its health route also
opens the database. This prevents idle database compute from suspending.

**Expected behavior**

An explicitly marked unclaimed application should keep its HTTP process
and sandbox provider plugins ready while leaving the database idle. A
successful signed claim should resume normal API behavior and background
processing. Claimed and self-hosted instances should keep their current
behavior.

**Steps to reproduce**

Start an empty Cloud-managed application and leave it unclaimed. Observe
database activity while repeatedly requesting `/api/health`. Before this
change, periodic queries continue without company data.

Related: #15153 reduces allocation during chat polling. This change
suppresses polling only for explicitly marked, empty, unclaimed Cloud
apps.

## What Changed

- Add `PAPERCLIP_CLOUD_WARM_STANDBY=1`. Check company emptiness once
after restoring the persisted Cloud runtime identity. Missing Cloud
configuration, existing data, or a persisted claim leaves normal
processing active.
- Gate recurring database pollers with an in-memory predicate. Keep
startup preparation and sandbox provider plugin loading intact.
- Serve unclaimed health probes without session or database reads and
report `warmStandby: true`. Serve standby pages/assets directly from the
UI router, bypassing session, bearer, tenant, and dynamic handlers.
Refuse API requests and all WebSocket upgrades before authentication can
query SQL or seed company data.
- Exit standby after the existing signed identity assertion commits.
Normal timers resume at their next tick; a restart restores the claim
even with stale provider variables.
- Document the marker, readiness semantics, rollout checks, and
rollback.

## Verification

- `pnpm -r typecheck` passed. A final server typecheck also passed after
adding tests.
- `pnpm build` passed.
- Focused standby, signed claim, restart, health, static/Vite routing,
hostname, HMR, and live-events suites: 72 passed after the review fixes.
Includes real HTTP upgrade admission before/after claim.
- `pnpm test:run` was attempted locally; both superseded runs were
stopped after encountering checkout/platform failures. A clean-checkout
rerun eliminated ancestor skill-directory lookup failures. The
company-skills/runtime-cache families encounter macOS
read-only-directory rename failures (`EACCES`); all three company-skills
failures reproduce on unmodified base `bf14f803d5`. The initial full run
also reported one native runner API test failure; an isolated comparison
on both revisions was blocked by local embedded PostgreSQL startup
failures. The full [Linux CI
run](https://github.com/paperclipai/paperclip/actions/runs/37423263993)
passed on final commit `5016c415ea`, including all server and workspace
test shards, browser suites, typecheck, build, and release canary. This
is not a claim that the full local suite passed.
- Isolated full server with local PostgreSQL: after startup and
connection expiry, 70 health probes, 70 page requests carrying valid
synthetic tenant credentials, and 70 rejected WebSocket upgrades over 70
seconds observed zero app database connections. The signed claim
completed in 62 ms and normal polling resumed (475 database transactions
over 12 seconds). Restart with stale provider variables restored the
durable claim. The latency is local-only, not a provider wake
measurement.
- Apex review: **5/5** on `5016c415ea`, both earlier threads resolved,
no open recommendations.
- No live-provider test or production deployment was performed. An
actual database suspension/resume canary remains required before
enabling the control-plane switch.

## Risks

- Standby health reports HTTP readiness rather than current database
connectivity. The signed claim still requires a durable database write;
claimed health checks retain the SQL probe and 503 failure behavior.
- Pollers resume at their usual intervals. A suspended database may add
claim latency. Validate the real provider before enabling the marker.
- Startup preparation and sandbox plugins remain loaded. New plugins or
background loops must respect the same standby contract.
- The marker is off by default. Remove it or set it to `0` and restart
to roll back. No schema migration or claimed-workspace inactivity policy
changes.

## Model Used

OpenAI Codex, based on GPT-6. The exact serving snapshot and configured
context-window size are not exposed in this session. Assistance included
source review, TypeScript changes, command execution, and PostgreSQL
tests.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` OR (b) described the issue in-PR following the relevant issue
template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [ ] I have run tests locally and they pass — targeted tests pass; full
local suite limitations are documented above, and full Linux CI is green
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-06 09:11:25 -07:00
DottaandCodex 0798c77fde Secure Cloud canonical runtime identity (#12766)
Accept and persist Cloud-signed canonical runtime identity before activation, then route absolute self-URLs through the durable runtime identity provider.

Co-Authored-By: Codex <codex@openai.com>
2026-09-03 12:01:47 -05:00