Commit Graph
13 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 f71aa135a0 fix(server): release sandbox leases for runs stopped before runner attach (#21788)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents in a `sandbox` environment run each task in a paid remote
sandbox that an environment lease tracks
> - When a run stops, the control plane must release that lease, and
terminating an agent waits until every lease of the agent is released
> - In a load test, a run stopped after its sandbox was ready but before
its runner started never released its lease, so the sandbox kept running
and the agent stayed in "terminating" forever
> - In the same test, terminating an agent whose failed run had a
scheduled same-run retry waited about 18 minutes, until the stale retry
time
> - This pull request makes lease release finish for every stop window,
records any release that must wait with a retry time, adds a hint-driven
safety net, and makes termination cancel a pending native retry at once
> - The benefit is that a stopped run never leaves a paid sandbox
running, and agent termination finishes in seconds without manual repair

## Linked Issues or Issue Description

**What happened?**
A load test with ten OpenCode agents in a `sandbox` environment stopped
its agents mid-run (pause, then terminate). Two defects appeared.
1. A run cancelled while it selected its provider pack (after the
sandbox was acquired and the workspace was prepared, before the runner
started) kept its environment lease `active` for more than 40 minutes.
The agent stayed in lifecycle `terminating`. The lifecycle worker
retried every pass and wrote `updatedAt` each time with no recorded
reason. Delete returned 409 "Complete termination before deleting the
agent".
2. A failed native run with a scheduled bootstrap retry kept its lease
until the stale retry time. Agent termination took about 18 minutes. The
run still showed "retry_scheduled" after the agent was terminated.

**Expected behavior**
A stopped run releases its sandbox lease within seconds. If release must
wait, the wait has a recorded reason and retry time, and a background
pass finishes it. Terminating an agent cancels a pending retry, releases
its lease, and reaches `terminated` promptly.

**Steps to reproduce**
1. Start a native run in a `sandbox` environment.
2. Stop it after the sandbox is acquired and the workspace is prepared,
before the runner starts.
3. Pause and then terminate the agent.
4. See that the lease stays `active` and the agent stays `terminating`.

**Paperclip version or commit**
Reproduced on `master` at 62033b9994 (load test build 2026.1106.0+2564).

**Relevant logs or output**
Root causes found in the code:
- `releaseEnvironmentLeasesForRun` returned with no record when
workspace retention refused a cancelled run. The refusal was permanent
when the run's native finalization owner had expired but was not
cleared. The owner is cleared only after `agentExecutionsHaveStopped`
sees every lease released, so the two waited on each other.
- `releaseRunLeases` silently skipped a lease whose prepared native
workspace was still "pending export", even for a run that never started
a runner and so never changed those files.
- The destroy path deferred on any busy warm session in the same
environment, also with no record.
- `stopInvocationsForAgents` returned on the first run that was still
live, so it did not release the leases of the other runs of the agent.
- The agent stop cancelled only `queued`, `running` and
`scheduled_retry` runs. A failed native run with a pending same-run
retry kept its lease until that retry fired.
- The lifecycle worker deferred a pending host step by 2 seconds every
pass, with no reason and no backoff.

## What Changed

- `releaseEnvironmentLeasesForRun` (`run-control.ts`):
- A native run with no runner launch evidence (no
`native.process_start_requested`, no runner event, no process id) skips
workspace retention and warm-session closing. No runner process exists
to fence, and no work ran against the prepared files.
- When retention is refused only because the run's native finalization
owner lease has expired, the release clears that expired owner
(compare-and-set on the exact expired owner of a cancelled run with no
result) and retries retention once. A recorded recovery then stops and
retains the sandbox, so the files stay recoverable.
- Every release that cannot finish now records
`metadata.releaseDeferral` (`reason`, `attempts`, `retryAt`,
disposition) on the active lease and publishes an `orphan-active-leases`
hint. The retry time backs off from 30 seconds to 15 minutes, then
retries hourly without end. The record keeps `failingSince` and the last
reason. A new Stop or terminate operation, or a server restart, triggers
an immediate attempt.
- `nativeWorkspaceExportPendingForLease` treats a cancelled run with no
runner launch evidence as owing no export.
- Safety net: `sweepOrphanedActiveLeases` (the existing hint- and
deadline-driven orphan-lease work) now retries the run's own release for
a due deferral or for a cancelled native run, after the native owner
lease has expired. The orphan-lease loader uses the stored `retryAt` as
the due time. No new timer or poller.
- Agent stop:
- It cancels a failed native run that has a pending same-run retry (or
an interrupted cancellation of one), through the existing
retry-cancellation path. A pause stamps `pauseStop` on those runs too,
so a budget resume can continue them.
- It releases the leases of every terminal run, and it skips a lease
whose deferral is not yet due.
- `agentInvocationStopBlocker` and `agentExecutionStopBlocker` return
why the stop must still wait.
- Lifecycle worker: a pending host or plugin step records
`pendingReason` and `pendingPasses` on the operation, and its retry
backs off from 2 seconds to a 60 second cap. When a release leaves a run
with no unsettled lease, the stopping agent's saved retry time is
cleared and the lifecycle queue is notified, so Pause or Terminate
finishes on the next pass.
- Cancelled-workspace preservation: a stopped run that never started a
runner has its prepared reference moved to
`nativeWorkspaceSyncReleasedUnchanged` when its lease is released, so
the preservation worker no longer selects it. The worker records the
same settlement itself for older runs, and waits on a live owner's exact
lease expiry.

## Verification

- Red on `master`, green on this branch (embedded PostgreSQL, fake
plugin sandbox provider):
-
`server/src/__tests__/heartbeat-lease-release-before-runner-attach.test.ts`:
agent pause then terminate, and board Stop, for each window between
sandbox acquire and runner attach (artifact discovery, pack select, pack
upload, runner launch, transport connect, session open). Each test
checks that the provider gets a stop or destroy call, the lease is
released, the run is cancelled, and the agent reaches `terminated`
within one extra worker pass. It also checks the orphan-lease safety net
for a leaked lease and a recorded deferral that the background pass
finishes when due.
-
`server/src/__tests__/heartbeat-terminate-pending-native-retry.test.ts`:
terminating an agent whose failed run has a bootstrap retry 17 minutes
away cancels the retry, releases the lease and reaches `terminated` at
once. The test covers an open task and a cancelled task.
-
`server/src/modules/agent-lifecycle/__tests__/pending-backoff.test.ts`:
a host step that stays pending records its reason, and its retry delay
grows to a 60 second cap.
- Related suites pass: run-control, recovery, orphaned active lease
sweep, environment runtime, agent lifecycle, native workspace
preservation and recovery, held comment wake.
- `server` typecheck passes.

## Risks

- A cancelled native run with no launch evidence now gets its prepared
sandbox destroyed instead of kept. The proof is the same launch evidence
that restart recovery uses.
- A launched cancelled run whose retention owes nothing is stopped and
retained, as before. A retention that throws (lock or write error, or a
live owner) now keeps the lease active with a recorded reason and retry
time, instead of a silent skip.
- Release now clears an expired native finalization owner of a cancelled
run. An old controller whose lease already expired then fails its next
owner check and stops, which is the normal lease rule.
- On the first start after deploy, the orphan-lease work releases leases
already leaked by cancelled native runs.
- Task state after pause then terminate: the run-scoped stop keeps the
task status (an open task stays open, a cancelled task stays cancelled),
and a run that held prepared files gets the existing workspace recovery
record on its stopped sandbox. No new state is added.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (code search, shell, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 12:54:23 -06:00
DottaandPaperclip b09f17fbfd fix(agents): repair queued setup through task connection cards (#20361)
## Thinking Path

> - Paperclip manages AI agents doing work for people.
> - Agent verification must finish before an accepted task can start.
> - A missing AI account default can prevent verification before a
provider starts.
> - Lifecycle policy treated that owned configuration refusal as a
temporary failure and retried each minute.
> - The accepted turn remained Queued without explaining the
prerequisite.
> - Park account configuration failures and reuse the task's existing
inline connection card.
> - Successful repair rechecks verification and releases the original
queued turn under its original authority.

## Linked Issues or Issue Description

**What happened?** An OpenCode agent selected the responsible user's
OpenRouter default, but that default did not exist. Verification retried
unchanged configuration indefinitely and its accepted task stayed
Queued.

**Expected behavior:** Preserve accepted work, explain the setup
prerequisite, and let the appropriate person repair it using the
existing task connection UI. Completion must verify the agent before
starting the same queued turn. Genuine pause, approval and access gates
remain effective.

**Reproduction:** Create an OpenCode Runner agent with a
responsible-user OpenRouter binding and no personal default; assign a
task while verification is pending. The base retries verification and
shows an unexplained queue.

Related: #18702 handles pending task connection recovery after account
repair. This PR handles the earlier agent-verification boundary. #20330
addresses a separate Codex governed-wait timeout and is already merged.

## What Changed

- Share owned account-selection failure evidence through a pure library
contract. Lifecycle policy parks those configuration failures with
fixed, redacted guidance; temporary infrastructure errors retain their
retries.
- Batch setup-state reads for unstarted queued turns. Distinguish manual
repair from automatic setup retries and preserve Stop, started/history
state and lifecycle holds.
- Reuse the existing `connection_intent` card, account selection and
completion APIs. Its controller-only `agentSetup` marker accepts only
never-started legacy turns; runtime tokens still require a running run.
- Address the card to the lifecycle verifier's account owner, preserving
the task's execution principal and private-task visibility. Attach
missing cards on admission, verification, startup and full recovery,
including retained retries and continuations. Repeated delivery and
publication remain bounded. A new queued turn supersedes the setup card
of a stopped turn; generic connection cards retain cross-run reuse.
- Recheck lifecycle verification after usable-account completion, then
release the existing turn through the normal readiness path. Preserve
decline, Stop, reassignment and changed setup-owner gates. Fence an
in-flight verification failure from overwriting a repair; add no
provider wake or recurring worker.
- Park owned account refusals during resume as well as initial
verification. Bind setup cards to the lifecycle operation, settle
obsolete deliveries, and keep explicit Retry guidance accurate.
- Reuse the router’s existing lifecycle pin for setup repair without
rerouting the task, advancing the pool cursor, or replacing its saved
binding. Legacy direct-account cards remain readable.
- Hide the incorrect Resume banner during setup, retaining genuine
manual/budget pause controls. Document the execution and retry contract.

## Verification

- Red: eight original behavior assertions failed on the base, including
the real selector's retry decision and the real batched queue
projection. Six owned selection cases failed classification.
Temporary-error presentation, different task/verifier users, retained
retry/continuation recovery and the setup Resume banner also have
failing-before/passing-after regressions.
- Green: 430 affected backend, lifecycle, shared-contract and UI tests
passed before the final review corrections. Subsequent focused runs
cover one inline card, real account selection/completion, duplicate
delivery/publication, verification before readiness, unchanged run
authority, private tasks, no extra provider wake,
decline/Stop/reassignment, company isolation, invalid runtime tokens,
transient retries and redacted diagnostics.
- Final setup/card and generic connection-service suites: 49 tests
passed. The stopped-turn case failed before the correction, then passed
with real account completion, the stopped source remaining cancelled,
and the replacement source remaining queued.
- After rebase, all 129 focused assertions passed. The local queue
suite’s 88 execution assertions passed repeatedly, but its
database-fixture teardown exceeded both 30-second and 120-second hooks.
An isolated queue case completed, and the setup/connection suites fully
passed. The original queue test and cleanup remain unchanged; broad CI
supplies the full-suite result.
- Server/UI typechecks, module boundaries, token gates and diff checks
passed. `pnpm verify --explain` requested broad CI because the
execution-contract document changed; no redundant full local suite was
substituted.
- Real source acceptance: macOS arm64, Node 26.4.0, isolated embedded
PostgreSQL and local board identity. Created an OpenCode/OpenRouter
agent and task through the API with no default. Admission retained one
unstarted turn and produced the existing inline Connect OpenRouter card.
The browser opened its credential form and closed it without entering
credentials. The same card/run persisted after server restart and
reload, with the setup label and no incorrect Resume. The browser also
reproduced Stop followed by a new queued turn retaining the stale card
on the prior head. On `12f609c98372fd1e50644028db75cfcc8deb210d`,
startup superseded the stale card and retained the same replacement
turn, with the original turn still cancelled and no provider execution.
- The earlier CI case-search failure also reproduces on exact base
`0e3fdebea8257bdff11573081c76b306f12db230` without this PR. Its repair
is now in master via #20371; this branch is rebased onto it and contains
no API-search changes.
- Staging's actual Runtime test returned “Connect an account and choose
your personal default” before provider startup. The original turn later
started independently on deployed code, then hit a separate
workspace-export failure. Automatic file recovery subsequently marked
that run successful; its task remains Blocked and the trial’s useful
output is unverified. This PR is not deployed; provider authentication
through its card and downstream publication remain unverified. No
account, credential, default, permission or binding was changed during
diagnosis.
- Previous revision review: Greptile 5/5 on
`12f609c98372fd1e50644028db75cfcc8deb210d`, with all four inline
findings addressed and resolved. [Full CI
passed](https://github.com/paperclipai/paperclip/actions/runs/42713494508)
on that revision: 59 successful checks, one neutral check and three
intentional skips, with no pending or failed checks.

- Latest revision `976bcae993805a8a2222395bbff9394f6791b424`: resume,
router and Retry-copy regressions failed before repair. Green: 92
lifecycle/setup/shared tests, 83 setup/generic-connection/router tests,
and 137 UI/shared tests. Server/UI typechecks, module boundaries and
token gates passed. [Full CI
passed](https://github.com/paperclipai/paperclip/actions/runs/42720731137)
on this exact revision: 59 successful checks and three intentional
skips, with no pending or failed checks. Greptile reviewed this head at
5/5; all six review findings are addressed and resolved. Real Stop/retry
acceptance retained the same replacement turn and current-operation card
through two actual server restarts, with no provider execution.

## Risks

- Known account configuration failures now require repair or explicit
Retry. Temporary failures keep automatic retries; accepted work and
authority remain intact.
- The projection adds one company-scoped batch read only for unstarted
queued turns. Card attachment examines at most 100 active candidates and
adds no timer, worker, schema or migration.
- The optional stored setup-card marker requires an aware reader during
rollback. No run/session/checkpoint identity format changes. Setup
presentation grants no access and cannot bypass verification or other
gates.

## Model Used

- OpenAI GPT-6 (Codex), with reasoning, tool use, code editing and test
execution. The exact serving model identifier and context-window size
are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-09 17:49:34 -06:00
DottaandClaude Opus 5.5 5a51a7685d fix(budgets): raise_budget_and_resume continues the work its budget stopped, once (#20144)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A budget hard stop cancels the scope's running work; the board
resolves it with Raise budget and resume
> - In the restart-chaos H1 scenario, the cancelled tasks did not
continue after resume and each needed a manual Retry
> - Resume is the board's explicit decision to continue, so the work it
stopped should continue once, as if the board clicked Retry
> - This pull request files that Retry automatically, keyed by the
cancelled run, once the run's stop has settled
> - The benefit is that a budget raise restores the work without manual
Retries, without repeating completed work, and without charging the
run's attempt budget

## Linked Issues or Issue Description

Refs #20143 (bug B; this PR builds on it), #20113 (bug A), #20101
(superseded: split into #20143 and this PR)

**What happened?**
Restart-chaos waves w-20261108T195944-s42 and w-20261108T210118-s9802
(H1). After `raise_budget_and_resume`, CHA-3's budget-cancelled run
(enforcement stop) and CHA-1's run (agent lifecycle pause from the
budget hold) stayed cancelled until someone clicked Retry.

**Expected behavior**
Resuming continues each task its budget stopped, exactly once, with no
attempt charged, an activity entry, and no repeat of work that already
continued. A run a person stopped stays stopped.

**Steps to reproduce**
Set a budget below spend while a task runs; let the stop cancel the run;
resolve with Raise budget and resume. Before this change, no new run is
queued.

## What Changed

- New table `budget_resume_claims` (migration 0451). One row per
cancelled run, outside the run's result JSON, so a Stop that settles
during the resume cannot erase it.
- `markBudgetCancelledWorkForResume` (called by `resolveIncident` before
the incident closes): claims every task run its budget stopped since the
incident opened. That covers runs budget enforcement fenced, including
stops still in flight, and runs the agent lifecycle pause stopped for
this budget (`pauseStop`, carried from the lifecycle operation's
`pauseCause` so a resume before the Stop keeps the link). Runs a person
stopped are excluded. Two indexed passes led by `(company_id, status)`.
- `deliverBudgetResumeContinuations`: once a claimed run's stop settles,
files the resolving board member's Retry (`requestTaskRetry`,
idempotency key `budget-resume:<run>`, with the member's session source
and the chat Retry callback). The Retry path checks, under the task
lock, that the run is still the task's latest and the task is open, so
continued or completed work is never repeated. Claim settlement and its
activity entry commit together; retryable refusals and errors retry.
- Delivery runs in the always-on budget-enforcement worker after each
pass's Stop delivery. Its schedule includes deliverable claims and the
earliest stale-claim expiry. The execution-continuation worker is a
backstop.
- `agent.paused` / `agent.resumed` logging is in #20113.
`budget.cancelled_work_resume_requested` (incident) and
`budget.cancelled_work_continued` / `_not_continued` (task) are recorded
here.

## Verification

- New `budget-resume-continuation.test.ts` (Postgres, real
`resolveIncident` → claims → `requestTaskRetry` → real `resumeTaskRetry`
into a queued run):
- each task continues once, including a blocked task; repeat delivery or
resolve saves nothing new;
- a stop still in flight at resume, whose terminal write replaces the
result JSON, is still continued;
  - a 30-task scope is fully continued;
- a lifecycle-pause stop caused by the budget is continued, and one
caused by a manual pause is not;
- continued work, done tasks, a run that finished before its stop, and
person-stopped runs (including ones enforcement also fenced) are not
continued.
- `lifecycle.test.ts`: the pause records its cause on the operation and
on stopped runs, including when the budget is resumed before the pause's
Stop. `budget-enforcement-worker.test.ts`: delivery after Stop delivery,
and a wake at claim expiry.
- 287 tests across the affected suites passed. Server typecheck
(`typecheck-layers`, low-memory): no errors.
- Not run: the H1 chaos rerun (Phase 3 runs it after merge).

## Risks

- Resume now continues budget-stopped tasks automatically. A board that
wants a task stopped can Stop it again; person-stopped runs are never
continued.
- A new table and one index (migration 0451).

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test execution). Extended reasoning
enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 22:56:28 -06:00
DottaandClaude Opus 5.5 96badeaa42 fix(lifecycle): never repeat a passed harness verification; settle a dead verifier's lease at once (#20030)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A new or reconfigured agent runs a harness verification (a hello
probe in a test sandbox) before it is ready, and its runs wait until it
passes
> - In the final Daytona restart chaos wave, a Codex agent's runs waited
about 417 s for their first sandbox, while boot took only about 32 s
> - Each control-plane restart killed the verifier. The next process
waited out the 120 s lifecycle lease and then booted a fresh sandbox.
Nothing recorded a configuration that had already passed.
> - This pull request records a passed verification by what makes it
valid and reuses it, and settles a dead verifier's lease on the shared
ownership path at once
> - The benefit is that a restart never repeats a passed verification,
and never leaves runs waiting out a dead process's lease

## Linked Issues or Issue Description

**What happened?**
In the final Daytona chaos wave, the Codex agent was created at 09:55:46
UTC and its three runs were queued. Verification ran four times, in four
sandboxes, because faults K1, K4 and K2 each killed the verifying
server. After K2 the server was ready at 10:00:20, but verification
restarted only at 10:02:13: it waited out the 120 s lifecycle lease. The
first run sandboxes were created at 10:02:43, 417 s after launch.

**Steps to reproduce**
1. Create a Codex agent and assign it a task at once, so the task waits
for harness verification.
2. Kill the control plane (SIGKILL) while the agent is verifying.
3. The restarted server waits out the dead verifier's 120 s lifecycle
lease, then verifies again in a new test target. The task's first token
arrives about 130 s after the fault.

**Expected behavior**
- A passed verification of the same effective configuration in the same
runner image is never repeated.
- A verification is never restarted from scratch while its holder
process is alive. When the holder is proven gone, its lease is settled
at once.
- Runs gate only on a configuration that has never passed.

**Root cause**
`verify()` (`server/src/services/agent-harness-verification.ts`) leased
a fresh sandbox on every attempt and recorded nothing on a pass. The
lifecycle lease (`modules/agent-lifecycle/adapters/postgres.ts`) named
no holder process, so a restart could only wait for it to expire.

## What Changed

- Restart chaos TTFT driver: a `verify` phase, reported in its own rows,
and a `--verify-fault K1|K2|K6` option. The fault fires 1 s after
`verifying` is observed, so it interrupts a held lease. The seed keeps
each agent's creation body, so the driver can hire an identical agent.

- **Persisted pass:** new columns `agents.harness_verified_digest` and
`harness_verified_at` (migration 0448). The digest is an HMAC, keyed
with the instance's decision-signing secret, over:
- the fully resolved effective configuration: every value the test used,
including `cwd`, resolved credentials, and agent-over-environment
precedence;
- the selected AI account and its credential generation (connection and
grant ids and `updatedAt`, never the credential);
  - the environment and the runner image.

The key means the stored value cannot confirm a guessed credential
offline. For a sandbox, the runner image is an immutable image digest or
a qualified Daytona snapshot ID (`sandboxBinding`, now exported). For
local runs it is the server version. A mutable image or snapshot name,
or a local environment redirected to the managed sandbox, never reuses a
pass.
- **Server-only:** `normalizeAgentBaseRow` and
`redactAgentRowForResponse` strip the digest and its timestamp from
agent API responses.
- `verify()` still runs every account, policy and secret check, then
returns complete on a matching digest without booting a test target. It
records the digest on a pass before the lifecycle step completes, so a
restart in between reuses it.
- **Lease holder:** the lifecycle `claim()` stamps the claiming
controller process on the lease (`lifecycleOperation.leaseHolder`,
supplied through the lifecycle effects port) and clears the stamp with
the lease. The compare-and-swap release lives in the lifecycle module
(`releaseAgentLifecycleLeaseOfHolderInTransaction`).
- **Shared ownership path:** a new `agent_lifecycle_lease` kind with
`settleAgentLifecycleLease`.
- When the reconciler proves the holder gone, settle clears exactly that
lease (compare-and-swap on the holder), notifies lifecycle work, and
logs `agent.lifecycle_lease_reconciled`.
- A live holder is never taken over. The lifecycle state, operation and
progress are untouched.

## Verification

- `restart-verification.test.ts` (embedded Postgres, real lifecycle
store and worker, real reconciler pass):
- A verification held by a dead process is claimed and completed at once
after the startup pass, with the agent ready in under 10 s instead of
waiting out the 120 s lease. It fails without the new kind.
  - A live holder is never taken over.
  - The claim stamps the holder, and `defer` clears it.
- `agent-harness-verification-preflight.test.ts`:
- A second `verify()` of the same configuration allocates no target. A
revoked account still fails before reuse. A changed model, `cwd` or
literal credential tests again. It fails without the reuse.
  - A failed verification records nothing.
- Ownership report, all lifecycle module tests, remote, provider-API and
unit harness verification tests, and crash-point call sites: 13 files,
220 tests pass.
- Targeted local run: the TTFT driver's new `verify` phase (`--phases
verify --verify-fault K1`, Codex, 2 samples) hires a new agent, assigns
it a task at once, kills the control plane (K1) while it verifies, and
times fault to first agent token:
- Before (master server code): 129.8 s and 129.8 s. The restarted server
waited out the dead verifier's 120 s lease. Run-log TTFT from run
creation was only 4.1 s.
- After (this branch): 15.8 s and 17.2 s. The startup reconciler pass
settled `agent_lifecycle_lease` and verification re-ran at once.

## Risks

- **A stale pass:** the digest covers every resolved value, the selected
account generation, the environment and an immutable runner image, so
any change tests again. Rotating the instance signing secret invalidates
every digest, so each agent verifies once more.
- **Migration:** two nullable columns, no backfill. Existing agents
verify once more, then reuse.
- **Settle:** it runs only on a `gone` proof, compare-and-swap on the
exact holder. It never changes the lifecycle state.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file editing, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 08:10:40 -06:00
Dotta eb7c7b083a Recover Muse verification after contact, restart and access restoration (#19909)
## Thinking Path

> - Paperclip manages AI agents and their work under company access
rules.
> - Muse readiness uses a saved connection, receiver contact and an
authenticated challenge reply.
> - A failed challenge write could leave a saved contact with no later
recovery trigger.
> - A server restart or restored access must recover that contact under
current authority.
> - This pull request uses the existing delivery coordinator and saved
Muse state for recovery.
> - It keeps the normal receiver schedule, authorization checks and real
deadlines.
> - It lets eligible verification recover and lets idle verification
stop work.

## Linked Issues or Issue Description

**What happened?**

The first Muse receiver contact could persist while readiness challenge
creation failed. The contact then had no durable retry or startup
recovery path. Later membership, lifecycle or feature restoration could
also leave pending verification without a notice.

**Expected behavior**

Retry a temporary failure through the existing worker. Recover pending
contact after restart. Recheck the current company, agent, operator,
credential and binding generation before issuing a challenge. Stop
verification work while it is ineligible or complete. Do not revive
revoked credentials.

**Steps to reproduce**

1. Use the normal invitation and OAuth registration, PKCE and token
flow.
2. Send one receiver contact and let its normal persistence schedule
run.
3. Fail the first readiness challenge operation before it commits.
4. Observe whether the existing worker retries without another contact
or a manual Verify action.
5. Restart the App with a new database owner. Check that it creates one
usable challenge from the saved contact.

**Paperclip version or commit**

The causal baseline test used the unchanged production implementation at
`90148317b2a447794ff29a4acae5648ad11a9770` plus its normal fixture. The
accepted positive test used `9d383041028776ad8469ac4af3fe14ba6c369425`.
The initial published head was
`fe518c9b72504daf6374225987f24d45bc7bcf75`. The lease/auth correction
was locally qualified at `cfd515a207416097fe8b586277d70c4fd4a679fa`. The
current candidate is `6efe722e7eb1e22acd466125c7b4a627611d0273`; its
added Cloud actor fixture correction passed the full normal owning
program and all 35 tests. Its current normal build, lifecycle owner
controls, healthy-wait and restoration measurements are described below.
Earlier results below are bound to their stated revisions.

Related public changes: Refs #19819 for original-principal
authentication. Refs #19750 for the transaction-owned external feature
read. These are related foundations, not duplicate fixes. The current
Muse title and head-branch searches found no duplicate
contact-verification recovery PR. Related maintenance work is #17000 and
terminal assignment settlement is #17793.

## What Changed

- Add a typed, exact-binding verification notice through the existing
Muse delivery queue.
- Register the first-contact notice before its durable write in the same
transaction.
- Recover pending contact from saved state at startup. Retry temporary
failures through the existing scheduler.
- Recheck current feature, lifecycle, credential and access state after
commit. Park unavailable authority without recurring verification scans.
- Add narrowly scoped notices to the existing access, lifecycle,
configuration, company, budget and settings write owners.
- Save a healthy lifecycle verification wait in the existing operation
JSONB. Keep generic unresolved-work and error retries unchanged.
- Commit authenticated confirmation and its lifecycle recheck in one
transaction. Retain a raced recheck under the active lease, and recover
saved waits once at startup.
- Retain a committed lifecycle recheck while an expired attempt still
owns the row, so its late wait cannot park accepted confirmation.
Replacement claims keep their existing owner fence and fresh fact read.
- Keep the receiver gap, persistence and per-segment contact limits
unchanged.
- Add real App OAuth, failure, retry, restart, challenge and quiet
evidence. Add two normal-route restoration tests.
- Document recovery, reconnect requirements and the existing
connected-receiver sleep restriction.

## Verification

- Causal baseline: the real invitation and OAuth flow saved one contact.
The first challenge operation failed. The normal test failed because it
observed one verification attempt instead of a retry. No setup or
cleanup failure was counted as this regression.
- At `9d383041`: normal own Shared/SDK, DB with migration 0441 assets,
Runner TypeScript core and surfaces, module boundaries, and all seven
normal App programs passed. One new redundant role comparison first
failed typecheck; that diagnostic remains recorded and the corrected
full App gate passed.
- At `9d383041`: all 17 maintenance component tests passed with zero
skips. All original deadline, rollback, drain, terminal settlement and
startup controls remain.
- At `9d383041`: the one normal App case passed. Real OAuth authorized
one receiver signal. A retry occurred about 5.01 seconds after the first
failure. A new Db/App owner created one challenge, advanced revision 2
to 3, and accepted authenticated confirmation and mailbox ACK. Query
instrumentation had an active positive control. All measured query,
callback, session and verifier counters stayed equal across the
unchanged scheduled 15-second quiet window. Token revocation and
old-token/signal rejection passed.
- Known App, worker, HTTP and database owners closed without cleanup
errors. The tests used bounded local PostgreSQL and HTTP fixtures. They
did not call a live Muse model.
- Current integrated semantics: the full normal App driver completed all
programs. Only a new test callback had two undefined-value diagnostics.
An explicit defined-value guard corrected that test. The full 416-root
owning program then passed with empty diagnostics. A separate collector
incorrectly required an explicit child `noEmit` field; the normal
inherited setting was verified without repeating the compiler. The
original failures remain recorded.
- The first restoration run failed both cases. Permission parking
recorded 120 main-pool queries without more verifier calls. The
lifecycle counter recorded two denials without their exact scope. These
observations remain unexplained at the individual-query/call level.
Source separately shows a lifecycle readiness retry every two seconds
while the real external reply is absent.
- At `fe518c9b`: normal Shared/SDK and DB producers, module policy, all
seven full App programs, and all 33 pure diagnostic controls passed. The
lifecycle owner file passed all 19 cases, including both
confirmation/park orders, rollback, duplicate notices, startup recovery
and authority controls.
- At `fe518c9b`: the normal OAuth failure/retry/new-Db reconstruction
case passed. Its new healthy-wait window measured 15,002 ms with
unchanged total activity before genuine confirmation; the existing ready
quiet and revocation checks also passed.
- At `fe518c9b`: both normal restoration tests passed in Vitest.
Ordinary viewer-to-operator restoration and Pause/Resume recovered the
same binding, grant and single contact; each committed one challenge and
advanced revision 2 to 3. All six parked, restored and revoked/disabled
quiet windows measured at least 15 seconds with active positive read
controls and unchanged total queries, callbacks, sessions and verifier
calls. Both cleanups passed.
- The restoration wrapper still recorded failure: it required lifetime
lifecycle denial count zero, whereas the measured count was one before
the parked quiet window. The reviewed fixture explicitly permits a
current-authority denial from an earlier observation and requires no
committed challenge before restoration. Precise denial timing relative
to Pause was not serialized. The original wrapper failure is retained;
no rerun or retrospective explanation of the earlier failures is
claimed.
- Review exposed an expired-lease race. The added real-store baseline
test failed after the late defer restored `waitingForSignal`; no setup
or timeout was counted as the regression. On `cfd515a`, the whole
lifecycle owner file passed all 20 tests, including that case. The full
normal production declaration program (1067 roots), verification program
(662), and auth owning program (405) passed; all 1942 declarations
remained byte-identical.
- The original remote auth-session shard failed three tests because
their partial query adapters lacked the new binding query's
join/order/limit chain. A fixture-only correction preserved the real
notifier and existing authorization assertions. All 12 auth-session
tests passed locally.
- Complete remote inspection also found 33 failures in the 35-test Cloud
actor fixture with the same missing `innerJoin` method and downstream
incomplete-auth assertions. Its original full log and exact synthetic
Source comparison are retained. At `6efe722e`, the three-line fixture
adapter repair passed the complete normal 662-root test program and the
whole 35-test file with zero failures or skips. All existing lock,
authorization, settings, debounce and unfinished-auth assertions remain.
- The earlier healthy/restoration quiet measurements remain bound to
`fe518c9b`. Those steady 15-second windows did not exercise a
still-owned attempt passing its 120-second lease. The targeted
expired-owner regression qualifies that new boundary; it is not a fresh
measurement of the earlier windows.
- The initial `fe518c9b` remote gates failed the auth fixtures and
reported a Greptile 4/5 P1. The aggregate failure reported the
underlying shard. Those actual causes were fixed and qualified above.
- Current `6efe722e` remote gates completed with **57 successful checks
and three explicit skips**. The full latest edited Greptile summary is
**5/5**, explicitly reviews this exact head, and has no new finding. The
original P1 thread is resolved. All general, inline and nested review
pages were read.
- One current-head CI attempt timed out after 30 seconds in the Runner
evidence/privacy file, with 6,864 passes and one failure. That file and
its transitive inputs are byte-identical to the CI base. A separate
local diagnostic preserved every assertion and the original timeout, ran
the full normal Runner test project and the entire 41-case file, and
passed all 41 with zero skips. Its private bounded timing records
measured the target body at 74 ms. The diagnostic was then removed,
restoring the exact original test. This is local recovery evidence; the
original CI stalled await was not recorded and its cause remains
**unproven**.
- One retry of only the failed jobs on the unchanged head passed the
Runner support suite and dependent aggregate verification. The complete
second attempt and original failure logs are preserved. No test,
timeout, privacy guard or product code was weakened. This result does
not turn the original timeout into an explained failure or a diagnosed
flake.
- Full recursive typecheck, full `test:run` and a full release build
were not repeated locally. The normal full App programs, focused
producers and tests above passed, and the required current-head remote
gates passed. Final review is against the exact unchanged head; local
quiet remains separate from deployed suspension.

Reviewer commands use the normal project entry points: `node
server/scripts/typecheck-layers.mjs`, the whole
`server/src/services/muse-maintenance.test.ts` file, and both normal App
PostgreSQL files. PostgreSQL fixtures require explicit owned
fixture/admin URLs and normal migrations before imports. Run the files
serially with one worker. The private controller retains exact Source,
output, worker and owner-closure receipts.

## Risks

- Missing or early producer notices could lose a restoration trigger.
Notices stay in the transaction that commits the write. The worker
checks current authority after commit.
- A stale notice must not grant access. Binding generation, current
membership, company, agent and credential checks remain authoritative.
Revoked credentials require reconnecting.
- The change adds no table, migration, alternate task queue or periodic
verification scan. It adds two optional fields to existing lifecycle
operation JSONB. Generic pending/error deadlines, setup holds and manual
repair remain obligations.
- Confirmation can race with an active verification attempt. The park
compare-and-set must retain the durable recheck. Startup recovery uses a
fixed cutoff and keyset pages; it must not become another recurring
scan. Wrong company, stale version, pause and revoked credentials must
stay fenced.
- The connected receiver still has its documented tenant-sleep
restriction. Worker quiet is not proof of deployed suspension or
external wake.
- The normal challenge reply is sent by an authenticated protocol
fixture. Live Muse background replies, cost/usage, full Muse lifecycle
and full hosted Dot behavior remain unqualified. Partial Dot profile ACK
and unsigned-device cutover stay withheld.
- Restoration coverage for the additional company, budget, configuration
and feature producers remains a separate qualification obligation. This
pull request does not mark the whole hosted area complete.

## Model Used

OpenAI `gpt-6.1-sol` authored the implementation and tests with local
code and tool use. A GPT-6-based Codex coordinator reviewed the changes
and owns integration. Its exact underlying model ID is not exposed.
Context window sizes are unavailable. No context size is inferred.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-11-07 16:41:15 -06:00
Nicky Leach 53937be283 fix(agents): select environment plugins only for agents on that environment (#19173) 2026-11-04 20:59:02 -08:00
DottaandPaperclip 1f55073f96 fix(cursor-cloud): support API-only controllers on managed stacks (#18310)
## Thinking Path

> - Paperclip helps people manage AI agents at work.
> - Cursor Cloud hosts execution behind its API.
> - Product setup still expected a Paperclip Computer.
> - Managed-only tenants therefore rejected valid setup.
> - This admits the validated API-owned controller.
> - Authority, Stop proof, budgets and cleanup remain enforced.

## Linked Issues or Issue Description

Refs #18068 and #18069; both dependencies have merged.

**What happened?**

Managed-only setup still requested a Computer. Live tests exposed
missing Cursor Stop proof and duplicate replay.

**Expected behavior**

Use Cursor’s API controller. Verify provider/controller termination and
replay each log identity once.

**Steps to reproduce**

Enable native Runner/managed-only. Configure Cursor model/repository/key
without a Computer, then Test, finish setup, run and Stop.

## What Changed

- Derive API-only placement from validated native Cursor configuration;
reject placement/command/workspace/home overrides and keep
Kubernetes-only execution closed.
- Skip Computers/checkout/workspace realization. Keep scoped tools,
completion, private staging, budgets, leases and recovery. Persist
placement; reject contradictions.
- Align Cloud cards, Test, Save and Finish setup; hold unready drafts.
Legacy keeps its Computer, credential and policy checks.
- Verify Stop from exact API ownership, audited intent,
provider-terminal proof, no Computer/process ownership and controller
retirement. Recheck immediate/deferred retirement; wake existing
recovery on durable evidence.
- Clear warnings when the actual stopped target later has authoritative
confirmation. Keep missing/other-run negatives and the existing
predicate.
- Publish safe status refresh after full-proof ACK. Retain bounded
native/physical cancellation history; add no poller or weaker proof
path.

## Verification

Core merged as `3910352c923a`; Product merged as `27d35314be8e`. Public
branch `5f86eb66b081` has tree `67344151d303`. Its exact-head Greptile
summary is 5/5; the fresh audit found zero unresolved threads and
complete pagination. The controlled workspace retry and required summary
passed. All 59 current-head contexts are terminal and clean. Prior
green-36 evidence keeps its identity.

[Current
CI](https://github.com/paperclipai/paperclip/actions/runs/41680511731)
first failed workspaces-a 1/2: the CLI company-import/export beforeAll
30-second runtime-info/child-PID/HTTP readiness wait expired. All 431 UI
files/4,714 tests and 285 other CLI tests passed (one skip). The cause
remains UNKNOWN. Child output records startup recovery complete, but
buffered CI prefixes do not establish its relation to the waiter
deadline; runtime-file/PID/last-health state was not captured. Both
unchanged local probes were RED. The second observed dependency
preflight still pending at the deadline; observer overhead was not
quantified. Neither proves the CI cause. After all functional jobs
finished, only the queued metadata summary was cancelled; the other 48
records stayed unchanged. Attempt 1 retains 46 successes, the original
failure, one skip and the cancelled summary. One same-head debug retry
was admitted at 18:42 UTC on November 3. Its workspace job passed at
19:23 UTC on the same checkout: UI 431 files/4,714 tests and CLI 33
files/286 tests, including company import/export. Test caching was
disabled. The new `ci / verify` summary passed; workflow attempt 2
completed SUCCESS. Required `ci / e2e` also passed. The final check at
19:53 UTC on November 3 found all 59 current-head contexts terminal and
clean: zero pending and zero failing. The 47 inherited completed/skipped
rows are not new executions. No code, assertion or timeout changed;
success does not explain the original failure or erase the local REDs.

CI checkout is `1ea9d8608ced`/tree `78cb1ec13d59` (parents
`2e37c73c05fe`, 5f86); both workflows use 2e37. Its 20 extra paths add
low-memory build/Rust task-memory. Scripts cohere; later matrix changes
only labels, with no newer debug-cache steps. Static audit found no
blocker. Supported API arguments omit the memory flag: no default
monitor/instruction changes. Flagged external launchers can change
Cursor instructions/digests and remain unqualified.

The merge added 41 master-identical paths, retained the green-36 inbox
hunk and found UI fixtures matching upstream. Startup/backup, recovery
projection, sleep diagnostics and binary backup change production.
Static audit found eight Stop/ACK owner blobs and 37 sleep-work SQL
templates unchanged, with sleep fences/single-flight accounting intact.
Exact precommit tree: 416/416 across eight server files, binary backup
1/1, server types PASS; repeated scopes are not summed.

The deferred-retirement P1 reproduced three real-DB/owned-process
failures; missing proof stayed unverified. Recovery passed 47 process
cases/33 proof negatives. Warning repair binds actual targets to
same-run proof. Validation passed 729 scoped UI cases, a separate
359-case thread run, 111 backend cases (84 CAS, 6 assembled, 21
activity/privacy), types/tokens. Proof validators/predicate are
unchanged; no new poller or increased timeout.

The d8d3 attempt stays FAILED: 45 passes, two workspaces-a failures,
derived verify failure and one skip. Both shards shared three failures:
two successful mocks omitted the account guard’s required userId. The
two-line fixture repair changed no guard, assertion or timeout. Three
failures reproduced; 54 access/auth tests then passed, including
account-change/denial negatives, plus UI types/tokens.

The 69285 attempt stays FAILED: 49 jobs, 46 passed, serialized shard 9
failed, its verify aggregate failed and one selection job skipped. The
shard passed 575/576 cases. Its aborted-snapshot case expected nine
computes but saw eight: a final HTTP 200 coalesced while seven ordinary
reads had not proved the aborted compute settled. Earlier CI and the
unmodified local baseline passed with a miss; no local RED is claimed.
The repair awaits the snapshot, allows exactly one extra GET if
coalesced, then requires a miss and retains nine computes/all HTTP429
assertions. No production code, hook, loop, sleep or timeout changed.
Six inbox and seven cache/admission cases passed locally, plus server
types; 30 unrelated cases were filtered. Green 36 passed 576/576 without
taking the extra-GET branch; local runs also missed that schedule.
Current CI event merge 1ea9 now passed 576/576 and naturally exercised
the former failure: coalesced (count 2, 1ms), then exactly one fresh
miss (count 1, 16ms), with strict nine-compute/HTTP429 assertions
intact. No instrumentation or retry was added.

Earlier compatibility passed scoped remote/inbox, routine, API-placement
and UI tests, plus types/migrations/snapshots/maps/tokens. The 89-path
adoption restored fleet routing without workflow overrides or added
skips. Each result retains its source identity.

Live qualification remains actual `9af1a5de42a2`: unchanged Artifact
grader PASSED 17/17 and active-Stop grader PASSED 21/21. All three
provider test resources were deleted canonically with GET404;
post-delete download returned the exact 27 bytes. Histories stayed
stable (Artifact 121 and Stop 67 canonical identities), with 21 company
runs, zero active work and no accounting/cancellation/reservation debt
or integrity findings. Saved settings reopened; no-prompt Test/Linux
admission passed with `claude-sonnet-4-5`. No new Save is claimed.

The first Stop sampling case stays FAILED on a 10.373-second gap.
Unchanged retry kept all 24 samples: 35.076 seconds quiet, maximum gap
4.584 seconds, plus a separate CANCELLED/artifact-absent read after 180
seconds. Initial grading rejected locator spelling before functional
checks; metadata changed only after same-file/inode/byte proof, with raw
evidence/oracles unchanged. Original Question/Plan/reconnect and
historical deleted-artifact results keep their identities.

Git trees verified 11 history-only conflicts and six earlier mainline
paths. Hosted 9af to the public BRANCH differs in 222 paths; 13,500
other mode/blob leaves match. Runner runtime implementations and six
observer/driver pins match 9af on that branch only; inherited
`drivers/acpx/runtime-host.test.ts` differs. No all-Runner/src,
CI/latest-master runtime or image equality is claimed. Later static
audits through `8ab89f41e499` found no concrete API-only blocker. That
base adds completed-history sleep proofs and an MCP cleanup migration;
protected Stop/ACK owners are unchanged. These later changes are outside
this CI and have no live qualification here. Host duplicate-card and
later recovery/UI repairs retain separate proof, not deployed 9af
identity. Direct-key coverage requires absent aiConnection; API-only
placement skips inherited remote-workspace disk-full paths, and
config-read gates match their parent.

Other retained failures: stale fixture 132/133; E8 48/135 with 87
reset-hook failures; Mac 17 SQL failures/skips;
title/startup/custody/Stop/deletion unknown causes and bounded
recoveries; superseded warning/50 duplicate identities; pre-repair
ACK/Core CI failures. Fixture RED/GREEN passed 7/7 browser and 121/121
shutdown tests with a failing broken-unsubscribe control; stable WSP
wrapper passed 128 after physical teardown failure. No full local-suite
pass is inferred. All paid attempts remain recorded; vendor dollar cost
is UNKNOWN.

## Risks

Experimental: Cursor independently acknowledges neither the effective
model nor instruction digest before the first prompt. Explicit values
are submitted and behavior checked live; unpriced provider dollar cost
stays UNKNOWN.

The placement exception applies only to validated native Cursor API
execution. Other local work remains denied under managed-only policy.
Missing/stale Stop or cleanup proof stays pending; damaged archives
remain failed. Explicit Legacy keeps its existing policy owners.

## Model Used

OpenAI Codex/GPT-6 family: tools, execution, browser checks, delegated
review. API ID/context window unavailable.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used and the limits of available model
metadata
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have linked public issues or described the issue in this PR
- [x] I have not referenced internal or instance-local Paperclip issues
or links
- [x] My branch describes the change and contains no internal ticket ID
- [x] I have run the scoped local tests reported above; retained
failures are stated separately
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation
- [x] I have considered and documented the risks
- [x] All required CI gates pass on the current head
- [x] Fresh Greptile is 5/5 with no unresolved findings on the current
head
- [x] I will address all reviewer comments before requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-03 13:55:39 -06:00
Devin FoleyandPaperclip 81ecaf0654 fix(server): bound failed lifecycle verification (#18383)
Failed lifecycle verification could allocate another test environment every minute indefinitely. Count failed host verification tests separately from claims, persist a repair wait for confirmed configuration failures or three generic failed tests, and clear it through authorized retry or execution configuration changes.

Preserve typed temporary execution-target failures through adapter checks so timeouts, transport failures and interrupted probes keep automatic recovery. Retain existing readiness, authorization, termination and cleanup gates. Document the policy and add real PostgreSQL persistence and actual OpenCode adapter regressions.

Verified 398 focused tests (one Windows-only test skipped on macOS), repository-wide typecheck and build, structural checks, fully green current-head CI and Greptile 5/5 with no unresolved comments.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-30 15:29:43 -07:00
DottaandPaperclip f4f02880ca fix(server): wake failed retry notices from committed changes (#17649)
## Thinking Path

> - Paperclip keeps exact failed chat retries in durable action and wake
receipts.
> - Deferred and declined retries owe safe notices to their original
conversation.
> - Periodic publication maintenance repeatedly inspects those receipts
when idle.
> - Committing receipt, authority, lifecycle and lease changes already
identifies when eligibility can change.
> - A scoped owner reconstructs durable obligations, drains bounded
candidates, retains contention, then becomes quiet.

## Linked Issues or Issue Description

This existing failed-retry notice slice follows #17647. It removes the
failed-retry notice duty from periodic publication maintenance while
preserving the explicit manual API. Existing action, receipt,
publication and source rows remain authoritative; no new table or
scheduler is added.

**Problem:** Removing the scan alone loses notices after real admission,
cancellation, early refusal, requester or agent restoration, and
relevant lease settlement. Millisecond cursor truncation can also repeat
denied history.

**Expected behavior:** Committed supported writes wake the scoped notice
owner, which uses current authorization and precise cursors. Real
retries survive failure, contention, restart and drain; settled work has
no recurring SQL.

## What Changed

- Extract failed-retry projection under the existing authorization
owner. Preserve current receipt/source authority and provider transport
authorization.
- Register bounded scoped work using the existing delivery
coordinator/scheduler. Preserve exact PostgreSQL cursor precision,
contention and a wake arriving during reconciliation.
- Extend the shared transaction-aware notification boundary to
failed-retry notices. Cover admission, conditional denial, early
refusal, requester changes, and agent lifecycle transitions to an
invokable status. Two shared receipt hooks and their budget fixture
correction already landed in the preceding inbound slice; preserve them
once.
- Notify when relevant native lease cleanup completes or a reusable
lease is retired, preserving claim fencing and caller-owned
transactions.
- Avoid the discovery scan identified in #17647: known task scopes
require zero discovery SQL; company/startup discovery uses a single
DISTINCT projection from the same candidate predicate. Bounded current
authorization still runs afterward.
- Remove only this duty from periodic publication maintenance, retain
manual behavior and startup reconstruction, and add real
producer/lifecycle/query-budget regressions with an audit.

## Verification

Published head: `fa95a1f2fb9d546aa62dbd3b43aa714f5c88bd35`, targeting
master after #17647 merged at
`1aa3a21ac41f602977638cf0be875b01a243e69f`.

- **42 distinct current combined-master cases pass:** 21
retry-notice/native-lease/restored-input producer cases (1,651 skipped),
plus all 21 app/startup and task-drain admission-release cases. Server
typechecking, build and module boundaries pass. The final follow-up adds
acceptance documentation only; all 17 owned server source/test hashes
are unchanged.
- The prior **404-case** matrix passes at prepared revision
`c2c53b839497be6f5e28aa2cd5ba50cfd1d76d4a`: 259 across six complete
consumer/lifecycle/environment suites, 20 notice/lease cases, 53
ordinary-notice and 72 Retry/budget cases. Those chat selections skip
1,651/1,618/1,599 cases. A subsequent 21-case combined producer check
passes after the parent restoration repair. These earlier runs retain
their revision scope.
- The normal two-parent integration
`43d0f84ea56845b8fc4946a7533ecbf7f07b679a` preserves the complete
19-path owned binary patch and all 23 unrelated master paths/deletions.
The semantic merge base is the verified landed parent head
`41f7e2c9e351efa6cfa5ef699c0e69df7425012b`; squash ancestry otherwise
causes historical conflicts. Both parent commits and the exact resolved
tree are verified; no rebase or force push occurs.
- Three scope-discovery regressions fail before the fix. The first SQL
projection fails with PostgreSQL 42702 (ambiguous company column), then
passes after using a distinct company alias. Known task hints have zero
discovery SQL; company/startup discovery has one query returning only
distinct task scopes. Existing bounded projection/current authorization
remain intact.
- Current producer tests cover commit/rollback, parked authority
restoration, real agent resume, early refusal, requester changes, native
lease cleanup/replacement, drain/restart/shutdown, precise cursors and
contention. Parent shared admission/denial/restored-input notifications
are preserved once. No polling backstop or storage is added.
- [The owner
audit](https://github.com/paperclipai/paperclip/blob/fa95a1f2fb9d546aa62dbd3b43aa714f5c88bd35/doc/plans/2026-10-25-failed-retry-notices-sleep-audit.md#final-landed-parent-acceptance--october-28)
records current commands, fixtures, failure history and limits. Fresh
exact-head remote checks and Greptile 5/5 are pending; old-head reviews
do not count.

The owner audit preserves prior failures and their original revision
scope. The old remote head's Runner receipt timeout, cancelled gate, and
review error are not current passes; fresh exact-head checks and
Greptile 5/5 are required. No full local repository suite, live provider
tenant, external wake alarm, physical suspension or savings claim is
made.

### Final remote gates

Exact published head `fa95a1f2fb9d546aa62dbd3b43aa714f5c88bd35` has
Greptile 5/5 with no actionable findings, successful current main CI and
ready-triggered security scan. The final dependency/quality review
workflow `40747725096` passed after waiting for a GitHub-hosted runner.
No Ponytail verdict is present.

Two automatic main CI requests and two identical-head quality-review
requests were created one second apart. The older main run was
automatically cancelled; only the proven duplicate, unstarted quality
request was cancelled manually. Their same-head, same-workflow
replacements both finish successfully. The merge guard verifies these
replacements, all current checks and current required-status integration
rules before excluding the superseded cancellation contexts. No actual
failing review or current check is waived and no protection bypass is
used.

## Risks

- Process-local signals cover supported writers using this boundary, not
independent SQL or other replicas.
- Lease completion now reads/locks the exact row before updating.
Reusable lease retirement wakes company scope because its previous issue
can differ. Claims and transaction ownership remain enforced.
- Broad discovery can inspect history inside PostgreSQL; DISTINCT
reduces returned rows and round trips, without claiming constant
database cost. Task hints are wake requests, so bounded reconciliation
handles already exhausted scopes.
- Native filesystem activation belongs to execution authorization and is
not an eligibility input for this notice projection. Other
notice/maintenance duties and Cloud sleep gates remain separate.

## Model Used

OpenAI Codex, based on GPT-6, with reasoning, local code inspection,
shell execution, and PostgreSQL testing. The exact serving model
identifier and context-window size are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-28 14:14:28 -05:00
Nicky LeachandPaperclip 835966dec8 Prepare agent-owned environments during hiring (#17761)
<!-- entire-trail-link-start -->
https://entire.com/gh/paperclipai/-/trails/1/changes/paperclip/1
<!-- entire-trail-link-end -->

## Thinking Path

> - Paperclip manages agents and their execution environments.
> - New agent setup saves an inactive agent before model connection
tests.
> - Some plugins need that agent to prepare its execution resource.
> - The wizard tested without the agent ID, while the setup hold
prevented preparation.
> - Environment changes also discarded the selected model account.
> - This change prepares the saved draft during setup and keeps normal
execution held.
> - Users can test the prepared resource before they finish hiring.

## Linked Issues or Issue Description

**What happened?**

An agent-owned plugin environment could not acquire a test lease during
hiring. Changing environments also returned the user to Connect and
cleared the selected account.

**Expected behavior**

Keep the selected account, invalidate its test, and prepare the saved
agent resource in the background. Test that same agent before finishing
setup.

**Steps to reproduce**

1. Start a new Claude or Codex agent and select a saved account.
2. Change to an agent-owned plugin environment.
3. Try to finish setup or reconnect the account.

Related: #17750 defers probes until after hire submission. This change
implements preparation during the wizard instead. The related PR search
and ROADMAP review found no other implementation of this setup
preparation path.

## What Changed

- Add the optional `requiresAgent` driver declaration. Require agent
lifecycle participation when it is set.
- Prepare saved drafts on creation or environment changes. Keep the
setup hold and wait in `verifying` without background verification
scans.
- Bind tests to the saved agent. Return a warning without allocating a
lease when the required agent is absent or still preparing.
- Preserve the account and configuration step when the environment
changes. Save the selection and invalidate the old test.
- Show active preparation and support authorized retries. Keep approval,
company, budget, and normal task admission controls.
- Document the contract and cover lifecycle, lease, manifest, and wizard
behavior.

## Verification

- Server and UI TypeScript checks passed.
- 112 wizard and setup helper tests passed. The shared manifest suite
passed.
- 123 lifecycle, policy, lease, and environment-test API cases passed
after the approval-gate fix. The diagnostic and driver suites and all 80
agent adapter route tests also passed.
- Token gates, module boundaries, map validation, and whitespace checks
passed.
- All 66 lifecycle/policy cases passed after the stopping-draft
permission fix. The new real-DB route case verifies creator success,
other-creator denial, and refusal to complete a hire while stopping.
- Full workspace typecheck and build were attempted. Both stop at the
unchanged Rust Runner step because Cargo is not installed on this host.
- CI on 43be3c93d passed all automated checks, including full test
shards, typecheck, build, Rust checks, browser suites, and canary
checks:
https://github.com/paperclipai/paperclip/actions/runs/40368174530.
- The full local repository test command was started. Two unchanged
workspace tests assume Git's initial branch is master; this host
defaults to main. Both fail in isolation under the host default and pass
with process-scoped init.defaultBranch=master. The broad run was stopped
after those confirmed fixture failures; CI supplies the full clean
check.
- Early new-test failures were fixture collisions and an unscoped sweep
assertion; the fixtures and assertion were corrected. One route-suite
load raced with SDK output regeneration; rebuilding first and running
the suite afterward passed.
- Initial CI exposed a route mock missing the new prerequisite lookup.
The fixture was updated and a real HTTP no-lease case was added; all 54
cases in that route suite pass. Security review identified allocation
before required approval; the new approval gate and its tests address
it.
- CI on 50b5a08af reported 32 of 33 expected calls in the unchanged
cleanup-history scheduling test. Its isolated case passed locally
without cleanup-code changes; the original failure remains recorded and
the final CI run passes on the current head.
- Standard Greptile reviewed 43be3c93d at 5/5. A subsequent deeper
review identified runner selection, stale detail reads, error retention,
and test file size issues; 70343387b addresses all four with resolved
threads. Standard review returned 5/5 on that commit and again on
rebased head fda42c5ab. The rebase preserves master's new managed
environment skill declarations alongside requiresAgent. Full CI on
fda42c5ab passes: 55 successful checks, two intentional skips, no
failures. Run:
https://github.com/paperclipai/paperclip/actions/runs/40405121641.
- After those fixes, 133 UI tests and 61 lifecycle/policy cases pass;
all eight focused UI regressions fail against the previous
implementation and pass with the fix. Token, map, and whitespace gates
pass. The stale-read fixture initially omitted the setup URL because
navigation is mocked; following the URL activates the real detail query
and reproduces the race.
- UI TypeScript passes. This host's server dependency links now point to
a separate scratch checkout lacking requiresAgent, so the ordinary
server check reports that missing property. A no-emit check pinned to
this branch's shared declarations passes; it disables declaration
generation and supplies explicit local type roots. The first temporary
configuration lacked those type roots; the corrected configuration
passes. CI remains the full workspace verification.
- After rebasing onto 8373d7867, all 133 UI, 61 lifecycle/policy, and 44
manifest cases pass again, as do UI TypeScript, token, map, and
whitespace checks. The server no-emit check now hits three mismatched
SDK/Runner declarations from the scratch dependency links; these are
outside this change. Full CI builds the complete dependency set.
- Review cycle: the first Apex review scored 3/5 with four findings. All
were fixed together, the branch was rebased to resolve a schema
conflict, full CI and standard Greptile passed, and a second Apex review
returned 5/5 on fda42c5ab with zero new comments. All six review threads
are resolved, and the PR is conflict-free. Review:
https://github.com/paperclipai/paperclip/pull/17761#issuecomment-6422635718.
- No live provider resource was created and no deployment was performed.

## Risks

Plugins must opt in and ship a compatible declaration. Companies
requiring hire approval keep drafts paused and defer allocation and
verification until approval. Other companies prepare drafts during the
wizard under their existing agent-creation authority; the setup hold
still blocks ordinary work. Preparation uses existing lifecycle
ownership, retry, and cleanup paths. The open wizard polls only while
preparation or stopping is active. No database migration is required.

## Model Used

OpenAI Codex, based on GPT-6, with reasoning, tool use, and code
execution. The exact serving model ID and context window size are not
exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have described the issue in-PR using the bug report fields
- [x] I have not referenced internal or instance-local issues or links
- [x] My branch name describes the change and contains no internal
ticket id
- [x] I have run the targeted tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation
- [x] I have considered and documented the risks
- [x] All Paperclip gates are green — 55 successful checks and two
intentional skips on fda42c5ab
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-26 15:21:47 -07:00
DottaandPaperclip 30b753f614 perf(budgets): scope accounting and deliver enforcement asynchronously (#17046)
## Thinking Path

> - Paperclip manages AI agents and their paid work.
> - Budgets must prevent new work and stop reported overruns.
> - Cost settlement also ran serial lifecycle reconciliation for every
eligible company agent.
> - That made request latency grow with unrelated idle agents and
delayed results on stop delivery.
> - This change separates atomic admission and accounting from durable
asynchronous enforcement.
> - Scoped live usage detects overruns without writing provisional
ledger charges.
> - The result is constant query scope for ordinary settlement and
recoverable stop delivery.

## Linked Issues or Issue Description

**What happened?**

Settlement and budget checks entered company-wide lifecycle
reconciliation. A company with 1,000 unrelated idle agents performed
1,001 accounting transactions for one ordinary cost evaluation. User
requests could also wait for provider termination. Running usage
checkpoints did not enforce reported monetary exposure.

**Expected behavior**

Keep the final reservation gate authoritative. Commit charges, scope
blocks, and enforcement intent before returning. Deliver stop requests
after commit. Read only applicable policies and execution scopes for
ordinary charges. Stop long-running work when supported reported or
priced usage establishes an overrun.

**Steps to reproduce**

1. Create a disposable company with one company budget and idle agents.
2. Evaluate a charge below the budget with 0, 17, and 1,000 unrelated
idle agents.
3. Compare query counts, company accounting transactions, and duration.
4. Use the controlled partial-usage and delayed-stop fixtures to check
enforcement and request responsiveness.

**Paperclip version or commit**

The measured baseline is `6c9b87d0922f1ebfc43e8b0c36889ac5ba7b9837` from
`master`.

Related public work: Refs #16976, #2006, #3539, #13763, #14296, #16853,
#13075. These cover the affected service charge, cancellation, and
accounting paths. This change removes their shared reconciliation
overhead.

## What Changed

- Batch applicable policy exposure under the existing company accounting
lock. Preserve exact fractional money and zero-reservation dispatch
fences.
- Commit a durable final budget dispatch claim before provider I/O.
Retain a separate final issue/run ownership transaction through
synchronous handoff, without holding the company accounting lock or
awaiting provider work. Preserve native recovery lease checks and
duplicate handoff fencing. Browser handoff rechecks current policy; it
reuses a consumed rate slot only for the identical policy generation.
- Separate ledger spend, live observed spend, and effective held
capacity. Use the maximum of reservation and live observation for each
unfinished operation.
- Deliver budget cancellation through the existing coordinator.
Revalidate generations and persist queued/running stop fences before
external termination. Bound stop concurrency to four.
- Apply agent holds only to the affected agent. Project budgets leave
other projects running. Project real company transitions in bounded
lifecycle batches.
- Replace periodic broad reconciliation with durable pending work,
startup repair, receipt replay, and concrete retry/window deadlines.
Retain drain and idle-safety checks.
- Enforce accepted cumulative usage checkpoints without provisional
charges. Final accounting atomically replaces live exposure and settles
reservations.
- Add operator-gated budget spans, regression coverage, map updates, and
a real-PostgreSQL comparison harness. No schema or first-party telemetry
change.

## Verification

- Full workspace `pnpm -r typecheck` passes on the final ownership fix,
rebased onto `bbcf5dfe98`. Production `pnpm build` passes on final head
`4c5f4de8b1`. Map, module-boundary, token, and Storybook catalog checks
pass.
- Focused real-PostgreSQL coverage includes the 292-case rebased
accounting/dispatch run, 63 browser cases, 52 run-control/cancellation
cases, 42 policy-generation cases, 77 lifecycle/commentary cases, and
the exact Muse confirmation regression. Native executor unit coverage
passes all 768 cases. Final ownership coverage passes: 74
adapter/use-case cases, nine public regression cases, and the additional
deleted-project race. These exercise competing ownership, budget changes
and charges between phases, zero/nonzero duplicate claims, rollback,
recovery contention, read-only capacity rechecks, and controlled HTTP
startup after commit. Another 74 pure/startup/tracing cases pass. The
complete CI matrix and both 5/5 reviews passed on `1ac9b58`; its
external security scan identified a stale-owner handoff race. The final
fix restores synchronous issue/run ownership fencing after the
accounting transaction commits. Final head `8e84f55ce8` has fresh
Greptile 5/5 with zero new comments, Paperclip review 5/5, and a passing
security scan. The complete latest-head [CI
matrix](https://github.com/paperclipai/paperclip/actions/runs/39535189808)
passes, including budget, browser, Runner, build, and canary jobs. The
unchanged Runner close fixture passed on its sole same-head job retry.
- Broad local `pnpm test:run` invocations were stopped before required
source updates (rebase and review fixes). They are not claimed as full
passes. Final full-suite coverage comes from the complete latest-head CI
matrix; local focused coverage and the incomplete broad invocations are
reported separately.
- Reproduce performance with `pnpm --filter @paperclipai/server exec tsx
../scripts/benchmark-budget-enforcement.ts
--baseline-ref=6c9b87d0922f1ebfc43e8b0c36889ac5ba7b9837`; repeat with
`--reverse`. The harness uses disposable real PostgreSQL, 20 warm
samples per variant/count, and records raw timings, SQL/lock counts,
agent reads, unchanged agent state, and query plans. Final
production-source measurement revision:
`eff713a4f72c7f4b7a4002a701cb481d4bfccd95`; the extended harness is
committed in `4c5f4de8b1`.

| Unrelated idle agents | Baseline median / p95 | New median / p95 |
Baseline SQL / locks | New SQL / locks | New agent reads |
| --- | ---: | ---: | ---: | ---: | ---: |
| 0 | 1.80 / 2.33 ms | 2.99 / 3.88 ms | 5 / 1 | 7 / 1 | 0 |
| 17 | 15.53 / 19.55 ms | 2.24 / 2.60 ms | 90 / 18 | 7 / 1 | 0 |
| 1,000 | 978.35 / 1,790.43 ms | 2.16 / 2.52 ms | 5,005 / 1,001 | 7 / 1
| 0 |

- These measurements use local PostgreSQL 18.1 and Node 26.4.0 arm64.
They isolate the cost-evaluation core and complete admission path,
without hosted-region or provider latency. Prior-revision reversed-order
controls also preserved the constant query counts; the final
measurements above use the final production source.
- Complete reserve → final ownership gate → immediate offline provider
result is measured separately, with one responsible agent and one active
operation at each unrelated-agent count. Admission p95 baseline/new: 0
agents **11.54 / 29.77 ms**; 17 agents **9.31 / 27.62 ms**; 1,000 agents
**24.51 / 74.05 ms**. Baseline admission was already scoped; the
stronger final fence adds bounded overhead. New counts stay constant at
53 Drizzle statements, three transactions, two company-accounting locks,
and five agent-reading statements bound to the responsible agent. Zero
unrelated-agent reads, exactly one provider callback per sample, and no
leftover runs or holds. All admission scenarios stay below 250 ms. No
external provider calls are included.
- The partial-usage/delayed-termination fixture passed 20/20: receipt
persistence p95 13 ms, receipt return to observed durable Stop fence p95
28 ms, complete receipt-start-to-fence p95 50 ms, unrelated database
request p95 1 ms while termination was blocked. After releasing the
controlled termination fixture, acknowledgement p95 was 14 ms. These are
separately measured phases; fixture acknowledgement is not live-provider
termination latency.
- Models and provider deadlines are unchanged. No deployment, live
inference, or staging acceptance is claimed. A quiet 30-minute follow-up
waits for the merged revision on staging, then repeats classifier and
controlled budget-stop acceptance in staging, recording hosted
bookkeeping, queue delay, and actual provider acknowledgement
separately.
- Development failures retained: finalizer notification regressions
(fixed; 27/27 affected cases passed), invalid browser admission fixture
assumptions (fixed; browser coverage now passes 63/63), and first-head
lifecycle/dispatch CI regressions (reproduced and repaired). A
prior-head Cases E2E test hit its unchanged announcement-dismiss handler
race while opening a filter; the same journey passed on `1ac9b58`. The
external security review then found a stale-owner race between
accounting commit and provider entry; final verification covers a
competing run in that gap while retaining provider I/O outside the
accounting lock. Early PostgreSQL startup attempts hit local semaphore
pressure; serial reruns succeeded. The final-head Runner lane had one
failure among 1,823 tests: the unchanged `timeout-attachment` fixture
failed at its first process close after 6.28 seconds. The sole same-head
job retry passed all 1,819 active tests (four skipped), including that
fixture in 1.85 seconds. Three isolated local probes could not run
correctly because the Rust fake-provider fixture binary was absent;
cleanup masked the original startup failure. CI builds that fixture
through its normal lane. These probes are inconclusive and are not
counted as passes.

## Risks

- This changes accounting, lifecycle, and cancellation ordering. Durable
generations, source/sequence checks, ownership leases, and company
locking remain required for correctness.
- Provider stop acknowledgement can be slow or fail. Stop intent remains
pending until existing termination proof succeeds; cancelled row status
alone does not acknowledge delivery.
- Mid-run limits depend on provider usage reports or supported price
calculations. Providers that report only at completion cannot provide
exact mid-call detection.
- Signals are process-local wake hints for the supported
single-active-writer model. Durable state and startup recovery remain
authoritative.
- There are no migrations, new queue tables, or cached balances. Repeat
the classifier and controlled budget-stop acceptance after normal
rollout.

## Model Used

OpenAI Codex, based on GPT-6, with reasoning, tool use, code execution,
and parallel coding agents. The host does not expose the exact backend
model revision or context-window size, so those values are not asserted.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-21 07:23:00 -05:00
Nicky LeachandPaperclip 79b2bf78dd Create agent records when onboarding names are submitted (#17005)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Core onboarding selects an agent and connects its provider account.
> - The current flow creates the agent record after authentication.
> - Setup needs a stable agent identity as soon as the user submits its
name.
> - This PR saves a paused setup record on name submission and finishes
the hire on that same record.
> - The existing sandbox authentication and approval paths continue to
apply.

## Linked Issues or Issue Description

**Subsystem affected**

Core agent lifecycle, shared validators, and onboarding UI.

**Current behavior**

Onboarding creates the database record at the final hire step. Earlier
authentication and saved setup have no agent record.

**Proposed behavior**

Create a paused agent with a setup hold when the user submits its name.
Do this before adapter selection or authentication. Update that same
record after adapter selection. Reuse its ID on retries and saved setup.
Finish that record through the existing hire path.

**Reason and benefit**

Give setup a stable identity without coupling this refactor to computer
provisioning or a new execution backend.

**Breaking changes**

Incomplete setup now leaves an inactive agent record. Existing clients
can still create and hire agents without the new optional fields. The
sandbox authentication flow and API key support remain in place.

**Additional context**

Requested Core-only refactor. Related work: #15422 changes runner
defaults. This PR preserves those choices. No duplicate early-record
refactor was found in the issue and PR search.

## What Changed

- Save the agent when the user submits its name in both onboarding entry
points. Use a provisional adapter until the user chooses one. Reopen
name-only drafts at adapter selection.
- Add retry-safe setup creation and same-record hire completion. Keep
approval checks at final hire.
- Save and restore setup configuration in both onboarding entry points.
Show Finish setup for paused setup agents.
- Restrict setup changes to authorized board users. Block generic resume
until setup finishes.
- Add lifecycle, authorization, connection, and UI coverage. Update the
specification and feature maps.

## Verification

- Latest focused server run: 126 lifecycle and AI-connection tests
passed after the Apex fixes. An earlier broader run passed 301
lifecycle, instruction, adapter-validation, and AI-connection tests.
Earlier authentication, retry, cross-company, and direct-create
regressions also passed.
- Name-save verification: 336 tests across the wizard, naming dialog,
New agent, action-button, and wizard-step suites passed. Coverage
includes retry identity, repeated submission, closing during a save,
saved-model restoration, and a disabled process adapter.
- Shared, server, and UI TypeScript checks passed. The latest UI package
check and direct server TypeScript check passed. The server package
preparation command could not finish locally because Cargo is
unavailable; full build preparation runs in CI. Map checks, token gates,
and `git diff --check` passed; each pushed diff was scanned for
credentials and PII.
- All six onboarding browser tests passed together with installed Chrome
against a fresh throwaway instance after the Apex fixes. They include
numbered duplicate names, first-time Copilot without a connected
account, and Grok’s saved model. They verify early persistence in both
entry points, closing and reopening a name-only draft, same-record
adapter selection, sandbox sign-in, login reload, and failed-connection
gating. Provider responses are simulated; Core persistence is real.
- Regression coverage includes incomplete Copilot creation versus final
validation, Kimi settings through redacted responses, stale local
drafts, company switches during creation, saved environments, and
provider changes with personal/shared old bindings.
- Apex found five name-only setup gaps: numbered names, early Copilot
account validation, the Grok model default, same-name draft ownership,
and provisional runner restoration. The fixes preserve server names,
defer implicit account binding until hire, set Grok’s model, avoid
name-based reuse of unfinished records, and restore Auto for pending
adapter choices. UI, database, and browser regressions cover these
cases.
- Standard Greptile found a Grok provider mismatch in the new adapter
save. The save now uses ACPX with the Grok profile. Both the 129-test
dialog/setup run and a real-API browser regression passed after the fix.
The first browser attempt stopped on an ambiguous empty-roster button
selector; it now scopes the click to the toolbar.
- Moving the save point exposed a saved-model reset and timing
assumptions in the same-name and portal fixtures. The source choice now
preserves the restored model, and tests cover same-name recovery without
a second hire.
- Initial CI exposed direct-create compatibility, concurrent setup
effects, fixture assumptions, and a missing-fields assumption in the
provider-validation guard. These were fixed and verified locally. Review
findings were addressed with authorization, recovery,
credential-persistence, and restored-configuration regressions.
- The final desktop capture passed after host resource cleanup. A repeat
first failed before Core started because macOS had exhausted
shared-memory IDs. Standalone PostgreSQL initialization reproduced the
same error. Releasing only unattached control segments from this task’s
stopped test databases allowed the unchanged revision to pass. No Core
or system setting changed.
- Local database tests required port access outside the sandbox. Missing
declared development dependencies were supplied outside the branch. No
live provider sign-in was tested locally.
- [Full monorepo
CI](https://github.com/paperclipai/paperclip/actions/runs/39522434236)
is green on `82b82fec189a172a885d4dd565ae2f4a18c73efc`. Standard
Greptile scored 5/5 with zero new comments (check `118969424903`). The
subsequent Apex review also scored 5/5 with zero new comments (check
`118972866331`) on that same commit. All review threads are resolved.
- The separate Superagent Security scan returned neutral/inconclusive
because its pinned merge-base checkout failed or timed out. GitHub
returned 404 for check and suite rerun requests. This is not a
successful security scan; the CI workflow and other security checks
passed.

## Risks

- Users can leave inactive setup records behind, including after name
submission alone. These records cannot run until hire completes.
- Setup completion touches hiring and approval boundaries. Tests cover
retries, company isolation, creator permissions, and approval snapshots.
- An uncertain credential-save result retains its new secret reference
for retry. A confirmed client rejection removes only credentials staged
by that attempt.
- No schema migration or VM provisioning change is required.
Agent-computer integration is out of scope.

## Model Used

OpenAI Codex, based on GPT-6, with repository search, code editing, and
tool execution. The exact deployment ID and context window are not
exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] Apex Greptile is also 5/5 on the same commit as standard review
and green CI
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-20 18:08:57 -07:00
Nicky LeachandPaperclip 2dd9811e8b Own agent hiring and termination in a lifecycle module (#15631)
## Thinking Path

> - Paperclip manages AI agents and their work.
> - Agent creation, approval, pause, and termination change whether an
agent can work.
> - These changes currently use separate service and database paths.
> - Some agents need setup after their record exists and before their
harness can run.
> - This PR adds one module that owns those state changes and their
transactions.
> - The module records progress and retries incomplete work after a
server restart.

## Linked Issues or Issue Description

**Problem or motivation**

A hire can become active before its execution environment is ready.
Direct agent state writes also bypass one common lifecycle boundary.
Termination does not have a durable step for resource cleanup.

**Proposed solution**

Use agent lifecycle commands for hiring, approval, pause, resume, and
termination. Keep proposed hires pending until approval. Prepare each
approved hire, test its saved harness configuration, and then permit
work. Wait for required plugin work before completing a step.

**Alternatives considered**

A general database commit hook would expose transaction details to
callers. A timer alone would delay each new hire. This module owns each
command transaction and starts its worker after commit. A periodic scan
recovers work after a process stops.

**Roadmap alignment**

This change adds the lifecycle boundary for the existing agent and
plugin systems. It does not add a user interface or a provider
implementation. It replaces the lifecycle delivery approach in #15391.
Related proposals include #11475 and #13099; this change does not add
their external executor protocols.

## What Changed

- Add durable approval, preparation, verification, pause, resume, stop,
and cleanup states.
- Keep agent creation, lifecycle state, configuration writes, and agent
record deletion behind `modules/agent-lifecycle`. Keep validation,
credential rules, reads, onboarding, invitations, general approvals, and
company data deletion in their services.
- Commit a hire before the remaining onboarding or invitation work.
Serialize requests with the existing dedicated database connection
helper. Reuse the hire after a failed attempt.
- Keep accepted work in the run queue during setup. Require execution
and cleanup evidence before completing pause or termination.
- Retain the original credential owner when approving migrated hires.
- Start lifecycle work after each command commits. Use database leases
and revision checks for retries and late results.
- Add a generic required plugin method and capability. Store the
required plugin IDs for each agent. Track host and plugin completion
separately.
- Keep harness verification and credential rules in services. Supply
their results to lifecycle through explicit ports. Reject failed logins
and incomplete probes.
- Remove reverse imports from lifecycle adapters into services. Share
record queries and validation below both layers.
- Use one hire decision path for agent IDs and approval IDs. Resolve the
approval and agent change in one transaction. Expose policy
reconciliation on the command object.
- Save configuration and verification invalidation through one module
operation. Ordinary commands own their transaction. Restrict the shared
transaction integration to its listed callers. Reject identity, company,
accounting, timestamp, and unknown fields.
- Require completed termination before deleting agent records. The
company deletion service calls the module through a separate deletion
interface.
- Save an approved secret before applying its agent binding. A failed
binding leaves the secret approved. A retry reuses it.
- Remove old permission and metadata export files. Move hire
configuration rules out of lifecycle persistence.
- Report failed worker steps and scans with operation context and known
error codes. Omit provider messages, stacks, and configuration values.
Document ordinary application use and extension rules.
- Recover expired native owners only after execution stops. Retry
cleanup of failed retained leases. Wait for setup in the live smoke
script.
- Keep manual, budget, and company pause reasons separate. Guard
execution status writes with the ready state.
- Add a migration, boundary checks, regression tests, and a lifecycle
guide. Keep the current UI.
- Limit periodic policy checks to 100 eligible agents per minute.
Continue from the last agent ID. Exclude pending hires and agents in
termination or a final state. Keep explicit company and budget updates
immediate.

## Verification

- Latest head `ccc3c86b0` aligns the shared status update schema with
the PATCH route. All 15 shared tests, 104 server route/OpenAPI tests, 2
CLI tests, and shared/server/CLI TypeScript checks pass. The Greptile
comment is fixed. CI is running; no new Apex review was requested for
this small fix.

- Rebase `75708ac94` rebases onto master at `2eb5187e4`. The backup test
uses the shared 90-second timeout from master; the older timeout commit
was removed. The affected backup test passes.
- Agent budget edits now pass the target agent ID through enforcement.
All 50 affected tests, server TypeScript, and 11 boundary tests pass.
Both configuration update paths have regression coverage. CI is running
on the latest head. No new Apex review was requested for these small
fixes.

- Rebase `9b568666a` preserves the run-control extraction and
delivery-work startup changes from master. All 559 affected tests,
server TypeScript, and 11 boundary tests pass. CI and Apex review are
running on the final head.
- The policy scan fix passes 56 lifecycle and diagnostic tests, 24
policy and caller tests, and 11 boundary tests. A regression test checks
the page limit, interval, excluded states, and cursor wraparound.

- Rebase `fd4698750` passes all 611 tests in eight affected lifecycle,
route, and heartbeat suites. Server TypeScript and all 11 module
boundary tests pass. The company deletion rollback and concurrent hire
tests are included.

- Follow-up `00aac7646` fixes the failed decision deletion test and the
public update schema. All 126 affected server tests, 8 shared tests, 2
CLI tests, and 11 boundary tests pass. Shared, server, CLI, and UI
TypeScript checks pass.

- Server TypeScript passes: `pnpm --filter @paperclipai/server exec tsc
--noEmit`.
- All 11 boundary tests pass: `node --test
scripts/check-module-boundaries.test.mjs`. The repository boundary scan
also passes.
- All 68 lifecycle, failure diagnostic, and harness verification tests
pass. Tests reject forbidden fields through the root command, combined
transition command, and restricted transaction integration. They also
check invalid fields returned by configuration preparation.
- Failure tests cover host work, plugin work, claims, renewals,
deferrals, and scans. Log tests check known error codes and exclusion of
private error contents.
- All 355 tests in 10 existing caller suites pass. These tests cover
secret approval, pairing, pending hires, credentials, company import,
built-in agents, managed agents, instructions, and skills.
- The branch is rebased onto `origin/master` at `2eb5187e4`. The queue
guards and cancellation lock fix use the extracted heartbeat queue.
Agent cancellation uses the extracted run-control file. The source diff
scan found no secrets or private URLs.
- Earlier checks cover approval retries, rollback, late verification
results, policy holds, deletion rollback, concurrent company deletion,
and queue admission during setup.
- Before the rebase, CI passed on `00aac7646` and Greptile Apex rated it
5/5 with no open comments. CI and a new Apex review are pending for the
rebased commit. The branch has no merge conflicts.
- The full local build and native Dot suite were not run. The host has
no Cargo. No live provider test was run.

## Risks

- Background policy recovery takes more than one minute when more than
100 agents are eligible. Explicit company and budget updates still
reconcile immediately.

- The restricted configuration integration keeps an agent reference and
its dependent records in one transaction. It cannot change a budget or
invoke a lifecycle transition. A periodic scan starts pending setup
after that transaction commits.
- Secret approval and binding use separate transactions. A failed
binding can leave an approved secret with no agent binding.
- New approved hires remain unavailable until setup and verification
complete. Existing clients see `paused` during these steps. This PR does
not change the UI.
- Migration `0320_daily_onslaught.sql` maps existing statuses and pause
reasons. It does not create external resources or claim that an existing
harness passed verification.
- Use a coordinated server update for this migration. Older server
versions can write legacy status without the new lifecycle checks.
- A failed onboarding or invitation request can leave a committed hire.
A retry reuses that hire and completes the remaining records.
- Deletion requires completed termination. Company deletion can require
another request while cleanup is pending.
- Required plugins must make repeated calls safe and reject older
revisions. A disabled or removed required plugin blocks completion until
it is restored.
- A new hire can run a second harness test after the existing UI test.
Some adapters can charge for this test.
- No live provider test was run. Provider implementation and resource
backfill remain separate work.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, code tools, and local test
execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have described the issue in this PR with the feature request
field labels
- [x] I have not referenced internal or instance-local issues or links
- [x] My branch name describes the change
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation
- [x] I have considered and documented risks
- [x] Targeted local tests pass
- [x] All Paperclip CI gates are green on the new commit
- [x] Greptile Apex is 5/5 with no open comments on the new commit
- [x] I will address review comments before requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-09 09:05:12 -07:00