Commit Graph
6196 Commits
Author SHA1 Message Date
Dotta efa3382b21 test(e2e): fingerprint executed plugin CLI source and loader 2026-10-28 08:15:28 -05:00
Dotta 250ba62791 test(e2e): bind installed team to selected OpenCode runtime 2026-10-28 08:15:27 -05:00
DottaandPaperclip 1c6021a4e6 fix(evals): bind inline secret helper to actual consumer task
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-28 08:15:27 -05:00
DottaandPaperclip 0e4ebca61c fix(evals): bind approval wait to actual task workspace
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-28 08:15:27 -05:00
DottaandPaperclip dea44755b0 fix(evals): prepare prerequisite people through public membership routes
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-28 08:15:26 -05:00
Dotta 81987af644 Accept lifecycle-ready running workers in catalog readiness 2026-10-28 08:15:26 -05:00
Dotta 8cd9e8bc6b fix(evals): keep fast response initial observation on original clock 2026-10-28 07:26:09 -05:00
DottaandPaperclip 1c7cbdc25b test(evals): preserve artifact outcome without extra task status gate
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-28 07:20:35 -05:00
DottaandPaperclip 3dab66775d fix(evals): bind answer race completion and ordinary draft evidence
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-28 07:20:35 -05:00
Dotta af01c4f6d3 fix(evals): await fast response save before model readback 2026-10-28 07:19:59 -05:00
Dotta 9ea0c8a61d fix(evals): retain original clock for seven prepared human journeys 2026-10-28 02:31:43 -05:00
Dotta d282ae2eee fix(evals): share original observation clock across selected native phases 2026-10-28 02:01:44 -05:00
Dotta 81abf04b21 fix(evals): qualify owned OpenCode post-stop proxy cleanup 2026-10-28 01:32:20 -05:00
Dotta 15fb2318bf fix(evals): preserve human origin on consumed queue probes 2026-10-28 01:32:20 -05:00
Dotta b70439f387 fix(evals): keep deadline boundary timeout strictly positive 2026-10-28 01:32:20 -05:00
Dotta b0eff403b8 fix(evals): share original budget for instance and confirmation waits 2026-10-28 01:32:19 -05:00
Dotta e232200747 test(e2e): pin configurable auxiliary tests to requested model 2026-10-28 00:55:14 -05:00
Dotta 1e686fa0f3 fix(evals): bind routine failure evidence to selected profile 2026-10-28 00:53:18 -05:00
Dotta 78783dda57 fix(evals): bind routine history grading to observed provider 2026-10-28 00:47:54 -05:00
Dotta 765dd51e0f test(e2e): retain context custody across budget handoff 2026-10-28 00:37:55 -05:00
Dotta fa7304ae5e test(e2e): share original budget across company followups 2026-10-28 00:37:55 -05:00
Dotta 6eca7f6bcc test(e2e): verify returned historical artifact copies 2026-10-28 00:25:04 -05:00
Dotta 2c6d746015 fix(evals): retain contracted delivery negation in date grading 2026-10-28 00:25:04 -05:00
Dotta fefc475470 fix(evals): accept exact natural historical delivery dates 2026-10-28 00:25:04 -05:00
Dotta 8b29ca5489 fix(evals): enable real workspace view before lifecycle journeys 2026-10-28 00:21:59 -05:00
Dotta 81bba1dd0d fix(evals): scope design approval claims to contrast clauses 2026-10-27 23:00:30 -05:00
Dotta 89063699af fix(evals): recognize explicit design revision ordering 2026-10-27 23:00:29 -05:00
Dotta 22dd6a1280 test(evals): report effective project-defaults timeout policy 2026-10-27 22:36:30 -05:00
DottaandPaperclip d1f983e395 test(evals): distinguish team pricing from negated per-user charges
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-27 21:56:43 -05:00
DottaandPaperclip 9434442348 test(evals): reject waived required skills without confusing unrelated history
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-27 21:56:43 -05:00
DottaandPaperclip 96dcd15449 test(evals): reject modal contradictions of required plan features
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-27 21:56:43 -05:00
DottaandPaperclip 021c155682 test(evals): ground team requirements in actual skill identities
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-27 21:56:43 -05:00
DottaandPaperclip 913ffeec7c test(evals): accept substantive single-page plan paraphrases
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-27 21:56:43 -05:00
DottaandPaperclip 33f99c6c5a docs(evals): preserve the composed operating section heading
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-27 21:18:36 -05:00
DottaandPaperclip b797e1cc29 test(evals): read delivered task HTML with the authenticated fixture actor
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-27 21:18:00 -05:00
DottaandPaperclip 1ba267de60 test(evals): use production chat timeout within finite cell budgets
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-27 21:18:00 -05:00
Dotta 01c585d965 docs(evals): record refreshed merged-baseline chat pilot 2026-10-27 21:08:53 -05:00
DottaandPaperclip 0195fcb016 Merge pinned production for the post-custody chat pilot
Preserve frozen 244-scenario YAML intent and 191-capability coverage. Production baseline: 39e811c9c5.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-27 21:03:19 -05:00
DottaandPaperclip 39e811c9c5 fix(runner): prepare diagnostic proxy and reduce bundled skill queries (#17941)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native turns reconcile company skills and select verified remote
runtime packs.
> - Catalog imports repeat folder transactions for skills in the same
bundled category.
> - Diagnostic requests can also reuse an older image proxy that lacks
the phase collector.
> - This pull request reuses categories within one import and selects
the current proxy for fresh diagnostics.
> - Each import has its own category map, and recovery keeps its
admitted runtime identity.
> - The result is less preparation work and a runtime that can provide
the requested measurements.

## Linked Issues or Issue Description

Refs #17918.

**What happened?**

A hosted run retained `PAPERCLIP_OPENCODE_PERFORMANCE_DIAGNOSTICS=1` but
emitted no provider phase batch. Its saved pack authority selected an
older image proxy. The deployed server proxy contained the collector.
The image proxy did not. Bundled inventory reconciliation also repeats
folder transactions for skills in the same category. The paired local
benchmark exposes that preparation cost.

**Expected behavior**

A fresh diagnostic request must use a verified proxy that supports the
current collector. Bundled skill reconciliation should resolve each
category once per import.

**Steps to reproduce**

1. Use a protected image pack with the same OpenCode interface revision
and an older proxy hash.
2. Enable `PAPERCLIP_OPENCODE_PERFORMANCE_DIAGNOSTICS=1` for a fresh
remote run.
3. Observe that the current backend accepts the older proxy and cannot
emit the requested phase batch.

## What Changed

- Reuse each bundled category during one import. Every skill still runs
its existing validation, access lookup, value comparison, and write
path.
- Compare the installed proxy hash with the current build when a fresh
run requests diagnostics.
- Keep compatible-pack reuse for ordinary runs and exact admitted
identity for recovery.
- Cover selection through the real backend, including recovery after
diagnostics are enabled.
- Document runtime preparation costs and admission limits for diagnostic
requests.

The skill importer owns category reuse. The provider verifier owns the
artifact comparison. The native backend passes the resolved diagnostic
requirement at the existing admission seam. The changes stay in those
existing owners and share one latency investigation and rollout.

## Verification

- Before the fix, the new verifier and backend cases failed because both
accepted a stale proxy.
- After the fix, 63 focused cases passed in 8.10 seconds. They cover
normal reuse, current diagnostic proxy selection, stale proxy rejection,
selected ACPX profiles, and admitted recovery identity.
- 110 skill and folder service cases passed against isolated real
PostgreSQL databases. Only cluster bootstrap used the existing local
PostgreSQL service. Queries, migrations, authorization, file handling,
and transactions ran on their real paths.
- A paired benchmark used the same immutable skill snapshot,
authenticated actor, 170-skill inventory, and 3.5 ms delay in each
network direction. It checked every delivered file and executable bit.
Warm median fell from 3.319 s to 2.482 s (25.2%). SQL calls fell by 81
in each matched sample, from 241–243 to 160–162. The bundled phase
median fell from 2.596 s to 1.721 s. The final measurement uses commit
`37eac8d8de`; only the skill importer differs from baseline
`7951455073`. These local results are not hosted acceptance.
- `pnpm -r --workspace-concurrency=1 typecheck` passed with the pinned
pnpm 9.15.4 and a 6 GiB Node heap. Earlier attempts exposed a subprocess
toolchain mismatch, a local IPC restriction, and a fixture type error.
Each was corrected before the passing run.
- Module boundary and whitespace checks passed.
- `pnpm build` passed for all workspaces.
- `pnpm test:run` did not complete. Embedded PostgreSQL initialization
failed after repeated startup attempts. The owned test subtree was
stopped after preserving the failure log. This is not a full local test
pass. On final commit `37eac8d8de`, all 53 successful check contexts
passed; two optional Storybook checks were skipped. Greptile is 5/5 with
no unresolved findings. The draft awaits merge approval.
- The catalog change has a measured local gain. The proxy change fixes
missing phase capture. Real hosted response latency remains above the
target and needs a new matched measurement after delivery.

## Risks

The category map belongs to one import and one company. The importer
still resolves current skill data on every invocation.

Fresh diagnostic runs can prepare a current verified pack, which can add
startup time. Admission can fail when matching runtime assets are
unavailable. Ordinary runs keep the existing fast compatibility path.
Recovery keeps the already admitted identity, even if diagnostics are
enabled later. The operator endpoint gate, collector limits, and closed
span fields are unchanged.

## Model Used

OpenAI GPT-6 through Codex, with repository inspection, shell execution,
browser tools, and Honeycomb queries. The session does not expose a
verifiable backend deployment ID or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-27 21:00:05 -05:00
DottaandPaperclip 2cd747d1b6 perf(server): batch connection install capability checks (#17945)
## Thinking Path

> - Paperclip manages AI agents and their access to external
connections.
> - The connection grants screen describes which agents can receive an
installation.
> - Each candidate currently repeats authorization for every retained
installation.
> - A company installation makes this work grow with the square of the
roster size.
> - This pull request checks retained targets once inside the existing
service owner.
> - The benefit is fewer authorization calls with the same scoped
capability results.

## Linked Issues or Issue Description

**What happened?**

The grants GET repeats connection preparation and retained-target checks
for each candidate agent. A local 100-agent company-install fixture made
50,300 SQL calls and 10,000 target lookups in one capability-description
operation. Open PR searches found no matching capability fan-out fix.

**Expected behavior**

The description must preserve retained-installation obligations and each
candidate's permission. It should evaluate retained targets once per
description. Writes must still authorize fresh.

**Steps to reproduce**

Run the committed benchmark with
`--baseline-ref=f6bc653f7f745adaafa26393852fb2f5a99256a4`. It creates
disposable synthetic PostgreSQL fixtures and loads the actual pre-fix
service from Git. It compares every positional capability, reason, and
operation.

**Paperclip version or commit**

Baseline `f6bc653f7f745adaafa26393852fb2f5a99256a4`.

**Deployment mode**

Built from source. Local real PostgreSQL proof. No hosted rollout.

## What Changed

- Add `describeAgentInstallCapabilities` to connection management.
Ordinary authorization and the batch share preparation, target checks,
and fallback completion.
- Use the batch only for direct saved non-AI connections and board
actors. Complex resources, delegated actors, and unusual saved target
IDs keep the ordinary evaluator. Fallback consumes the first preparation
once.
- Replace only the grants route's editable-agent loop. Keep
configure/use checks, identity filtering, audience disclosure, and write
authority on their existing paths.
- Add scoped-grant, retained-target, error-order, freshness, fallback,
full HTTP response, and real mutation regression coverage. Add a
reproducible benchmark and evidence notes.

## Verification

Real PostgreSQL 18.6 Linux Docker, Node 26.4.0. Two warmups and five
measured samples per arm, in alternating order. These are
service-component measurements with the same SQL-count instrumentation
in both arms. They do not measure the whole hosted endpoint.

| Fixture | Old median | Batch median | SQL calls old → batch | Target
lookups old → batch |
| --- | ---: | ---: | ---: | ---: |
| 100 agents, company install | 24.753 s | 246.39 ms | 50,300 → 503 |
10,000 → 100 |
| 100 agents, partial install | 6.339 s | 236.37 ms | 13,075 → 502 |
2,575 → 100 |
| 1 agent, company install | 5.13 ms | 4.82 ms | 8 → 8 | 1 → 1 |

All eight fixture shapes and every measured iteration had complete
positional equality. One-agent timing ranges overlap. Full ranges,
database settings, and limits are in
`doc/plans/2026-10-27-connection-grants-batch-verification.md`.

- **Passed:** `pnpm -r typecheck` and `pnpm build`, using pinned pnpm
9.15.4.
- **Passed:** 116 focused tests: all 83 connection-management
integration cases use real PostgreSQL and HTTP; 33 batch regressions
execute real policy over synthetic query transport. The integration
cases keep existing database creation, migrations, clients, services,
and assembled HTTP server, with disposable Docker PostgreSQL.
- **Passed:** complete grants GET parity across six human-role fixtures;
real install PUT succeeds before revocation, then fresh GET and actual
PUT deny after revocation without changing saved installs.
- **Passed:** a real PostgreSQL red/green regression for empty saved
target selectors, including candidate-specific decisions and later
reached infrastructure faults.
- **Passed:** independent old/new evaluator comparisons, including
recursive and delegated fallback, run-context reuse, continuation,
failure order, and principal/company isolation.
- **Local full-suite limit:** `pnpm test:run` was attempted and stopped
after native PostgreSQL fixtures could not initialize. An isolated
failed native case reproduced that startup error. The host cannot
allocate PostgreSQL SysV semaphores, including outside the sandbox. The
focused suites use an existing constructor seam to start disposable
Docker PostgreSQL. This is not a complete local full-suite pass.
- **Passed:** all 16 native-runtime cases that failed to start
PostgreSQL on the host, rerun with the disposable Docker provider. The
full local workflow remains incomplete.
- `git diff --check` passes. Greptile is 5/5 with no actionable findings
on final head `41dcf3ae029b6c60999ca6208136399b0af663d0`. All final-head
CI gates passed, including the full PR workflow. The two Storybook
checks are expected skips because this change has no UI changes.

Normal reproduction:

```sh
node --import ./server/node_modules/tsx/dist/esm/index.mjs scripts/benchmark-connection-install-capabilities.ts --baseline-ref=f6bc653f7f745adaafa26393852fb2f5a99256a4
cd server
pnpm exec vitest run src/__tests__/connection-install-capabilities.test.ts src/__tests__/connection-management.integration.test.ts
```

## Risks

Retained facts are observed once during one descriptive call. A mid-call
revoke or deletion can leave a stale displayed choice. The next
description and every mutation or human continuation read fresh
authority. The batch returns no reusable authorization context.

The bound applies to target evaluations in the eligible shape. Scoped
grant payloads and hierarchy reads can still grow inside each policy
decision. Complex resources and agent callers keep existing costs.
Hosted latency improvement remains unverified. There is no migration,
provider protocol change, new worker, persistent cache, or UI contract
change.

## Model Used

OpenAI Codex with code editing, execution, and test tools. Design, test,
and independent review agents used exact model IDs `gpt-6.1-sol` and
`gpt-6-astra` with xhigh reasoning. The host did not expose the primary
session's exact model ID or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-27 20:59:01 -05:00
DottaandPaperclip 38d08847b8 fix(runner): keep OpenRouter upstream keys in the parent proxy (#17949)
## Thinking Path

> - Paperclip manages agents and their work.
> - Paperclip Runner starts OpenCode with the selected provider and
model.
> - The native OpenRouter route passed the reusable upstream key into
the child environment.
> - A persisted-state scan found that key in an OpenCode database WAL.
The exact write operation is not known.
> - This pull request reuses the existing parent-held provider proxy for
that route.
> - The child receives a temporary loopback credential instead of the
upstream key.

## Linked Issues or Issue Description

**What happened?**

The native OpenRouter path passed `OPENROUTER_API_KEY` into the OpenCode
child. A real Product E2E persisted-state scan detected the key in the
child database WAL. The scan does not establish which operation wrote
it.

**Expected behavior**

The runner keeps the reusable upstream key in its own process. OpenCode
can call only the selected model through the session proxy.

**Steps to reproduce**

1. Start the OpenCode driver with an explicit or ambient synthetic
OpenRouter key.
2. Inspect the fake child configuration, environment, and persisted
provider values.
3. Send a request through the real loopback proxy and verify the
upstream authorization and exact model.

**Paperclip version or commit**

The regression is present on merged master
`813e808af462b6a6f9db38b6791e3bfe8465b51e`. This PR is rebased onto
merged master `d40f9ac377`.

**Deployment mode**

Built from source. The finding came from an AWS Product E2E job. The
regression tests use local synthetic servers and a fake child, with no
provider calls.

Related issue search found #16643 about general inherited environment
scope. This change addresses the narrower native OpenRouter path. No
duplicate custody PR was found. This PR must remain unmerged until
separately approved; it does not change the eval baseline or qualify a
new live result.

## What Changed

- Reuse the existing parent-held proxy for native `openrouter/` models.
- Remove the upstream key from the child environment and retain it in
diagnostic redaction.
- Keep the selected provider and exact model. Reject alternate-routing
fields before the upstream request. Use a session-lived loopback token.
- Preserve the effective explicit or ambient proxy bypass list when
adding loopback entries.
- Add explicit and ambient key regressions, upstream routing checks, and
child persistence checks.
- Document the boundary and its limits.

## Verification

- Final rebased head: full OpenCode driver suite, 64 synthetic tests
passed, and runner source TypeScript check passed. Original author run
had 61 passing tests before the new master tests.
- The two new explicit and ambient regressions fail on the old
implementation.
- Independent source review passed. The two new synthetic regressions
also passed independently. A separately filtered legacy pair hit fixture
selection/chmod ordering and is not claimed as a passing run.
- Tests use the real driver and proxy with a fake OpenCode child and
synthetic upstream transport. They check key exclusion, exact
URL/model/authentication, wrong-model and alternate-routing rejection,
proxy bypass preservation, redirect rejection, and proxy shutdown.
- The two advisory regressions failed before the correction: fallback
routing reached upstream, and ambient NO_PROXY was lost. The full
64-test driver suite and source TypeScript check pass after the
correction.
- No real OpenCode SDK/provider qualification was run. The original WAL
write mechanism remains unknown.
- Repo-wide typecheck, test:run, and build were not run locally because
local native/DB/model execution is excluded. Normal PR CI passed on head
`3e1903272ce65b92055ea70f8dc586d9aa5e0205`: typecheck/release registry,
general and serialized suites, runner static/Rust/Vitest checks, E2E
shards, build, and canary dry run. Greptile is 5/5 on that head.
Paperclip Review is neutral with no blocking findings; its two earlier
findings are fixed and resolved.

## Risks

The native OpenRouter route now uses the existing OpenAI-compatible
proxy transport. Synthetic tests do not prove every live SDK response
shape. The loopback token remains a credential, is redacted, and expires
with the session. This change does not claim isolation of every child
secret or persisted value.

## Model Used

OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) assisted source investigation,
implementation, synthetic tests, and review with tool use. No provider
model call was used for verification.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-27 20:57:19 -05:00
DottaandPaperclip e4b30163bb fix(evals): preserve the complete trusted controller artifact inventory (#17943)
<!-- entire-trail-link-start -->
https://entire.com/gh/paperclipai/-/trails/103/changes/paperclip/103
<!-- entire-trail-link-end -->

## Thinking Path

> - Paperclip manages real agent work.
> - Controlled Runner evaluations record model spend and traces.
> - A trusted build creates one portable controller with a complete file
inventory.
> - GitHub artifact upload omitted one hidden dependency metadata file.
> - The next job correctly refused the incomplete inventory before any
paid worker started.
> - This change preserves all inventoried bytes without relaxing
verification.

## Linked Issues or Issue Description

Refs #17889, #17929.

**What happened?**
[Run
40585138437](https://github.com/paperclipai/paperclip/actions/runs/40585138437)
failed at copied controller verification. Its artifact contains 294
files; its signed inventory expects 295. The only missing file is
`server/node_modules/protobufjs/node_modules/@protobufjs/pool/.npmignore`.
No extra files exist. Every paid worker was skipped and no model request
ran.

**Expected behavior**
The trusted controller artifact preserves its full inventory, including
hidden package metadata. The strict verifier rejects any actual change
or missing file.

**Steps to reproduce**
Build the portable runtime, upload the directory with the action's
default hidden-file exclusion, then verify the downloaded artifact
against the original receipt.

## What Changed

- Enable hidden files only on the fresh trusted controller code upload.
- Download that exact upload and run a real artifact-service round-trip
test before any private preparation. The test fails if the upload drops
hidden files again; it tests the actual transport instead of matching a
YAML flag.
- Keep the strict byte inventory and trusted source/receipt checks.
- Document why this code artifact differs from private attempt evidence,
which still uses the protected archive.

## Verification

- Exact retained artifact comparison: one missing `.npmignore`, zero
extra files.
- `actionlint .github/workflows/runner-protocol-live-evals.yml` passed.
- Protocol workflow security tests: 14 passed.
- The artifact-service integration test is explicit-only locally. The
controlled workflow supplies the real downloaded artifact, trusted
source and receipt digest.
- No product runtime, provider routing or grading code changed. CI runs
broader checks. Fresh controlled workflow qualification follows merge.

## Risks

Hidden files are allowed only in the code-only output of the
provider-credential-free build. Raw private evidence and credentials do
not enter that upload. The artifact verifier still requires every file
and SHA-256 from the trusted receipt.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, repository inspection and
code execution. The exact serving model ID and context window are not
exposed to this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-27 20:55:18 -05:00
DottaandPaperclip a8bd012635 fix(evals): apply reviewed lock corrections to reports and qualification images (#17937)
<!-- entire-trail-link-start -->
https://entire.com/gh/paperclipai/-/trails/98/changes/paperclip/98
<!-- entire-trail-link-end -->

## Thinking Path

> - Paperclip manages agents and records their work.
> - Product E2E campaigns measure real agent behavior and model spend.
> - A paid hosted run completed both cases and all ten model receipts.
> - Its report job could not install dependencies from the committed
lock.
> - The trusted checkout needs a reviewed dependency correction.
> - This change applies a reviewed lock correction before the frozen
report install.

## Linked Issues or Issue Description

Refs: #17928

**Description**
The report in [run
40576181788](https://github.com/paperclipai/paperclip/actions/runs/40576181788)
failed with `ERR_PNPM_LOCKFILE_CONFIG_MISMATCH`. Both paid cases had
passed. The worker and build jobs already restored the resolved source
lock, but the report did not.

**Steps to reproduce**
Dispatch the bounded daily campaign on the default branch with a stale
committed lock configuration. Workers use the resolved lock. The report
uses the stale lock and fails before report generation.

**Expected behavior**
The report installs frozen dependencies from a verified lock for the
exact trusted source. Evidence can be reported without repeating paid
work.

## What Changed

- Apply a reviewed lock correction from the trusted report checkout
before its frozen install. Use the same correction for the qualification
image build, which also requires a compatible frozen lock.
- Pin the original lock, correction and output SHA-256. The trusted
authorization job rejects unknown inputs before any paid work.
- Apply the same correction to trusted manual history publication.
Qualification image builds apply it only when their target equals the
workflow revision; other targets keep their own lock.
- Update existing Claude ACP and postgres patch identities and link the
declared yaml dependency. Existing package tarball integrity entries
stay unchanged; the declared yaml dependency adds exactly yaml@2.9.1
with pinned integrity.
- Keep runtime registry resolution and worker target locks out of the
credential-bearing report.
- Test valid correction, idempotence and changed-lock, changed-patch and
wrong-output rejection.

## Verification

- Focused workflow and lock-correction tests: 36 passed.
- `actionlint .github/workflows/runner-full-stack-e2e.yml` passed.
- Original worker evidence: 2/2 passed, ten final model receipts, zero
unresolved model requests.
- The original report was regenerated from retained evidence with zero
provider calls and saved privately with replay provenance.
- Broader repository checks run in CI. Prior full Product checks passed
for #17928.

## Risks

The correction is tied to the reviewed committed lock. Dependency input
changes need a reviewed correction update. No new registry version or
lifecycle script can enter the privileged report through this fix. The
original failed workflow stays failed. No root lockfile is committed.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, repository inspection and
code execution. The exact serving model ID and context window are not
exposed to this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-27 20:55:13 -05:00
DottaandPaperclip 0d22e27dc4 fix(runner): preserve closed native startup diagnostics (#17962)
## Thinking Path

> - Paperclip runs agents through a controlled execution boundary.
> - Native Runner starts ACPX providers through a TypeScript sidecar and
a Rust transport.
> - That transport hides arbitrary provider messages because they can
contain credentials.
> - Grok admission failures currently have no stable code. Other startup
failures can also lose their owning stage.
> - A real native startup failure therefore reports only
`classification=unclassified`, even when the ordinary subscription hello
test succeeds.
> - This pull request preserves fixed failure categories without
forwarding provider text or changing admission policy.
> - The benefit is a useful next diagnostic result before another
functional repair or retry.

## Linked Issues or Issue Description

Refs #17922 and #17942.

**What happened?**

A native Grok session failed before its first tool. Runner assets passed
verification. The Rust rejection discarded the cause because only closed
error codes may cross that boundary. The existing admission guard used
plain errors. Startup errors outside that guard could also lose their
stage.

**Expected behavior**

Report a fixed Grok admission reason, or a fixed startup stage when no
known error code exists. Keep credentials, provider messages, session
IDs, and paths out of the classification. Keep the failure and cleanup
effective.

**Steps to reproduce**

1. Have a Grok ACP fixture return a session without its native effort
acknowledgement.
2. Start it through the real ACPX client and pass the failure through
the sidecar classifier.
3. Before this change, the result is `acpx_sidecar_command_failed`. Rust
reduces that to `unclassified`.
4. The same loss occurs for injected sandbox, credential, and runtime
admission failures.

## What Changed

- Give Grok effort admission errors fixed codes for missing
acknowledgement, model or effort mismatch, session mismatch, conflicting
configuration, drift, and invalid protocol use.
- Record closed host startup stages in a private weak map. Preserve the
original error object, cancellation, and resource cleanup.
- Prefer existing known error codes over a fallback stage. Ignore
arbitrary provider-supplied stage properties and messages.
- Map only exact new codes to Rust diagnostic categories. Add real
ACPX-client and sidecar-pipe regressions with secret canaries and
unknown-code controls.
- Bind the changed shared source and new diagnostic modules in Copilot
profile 70. Preserve profile 69 and earlier history.
- Document the diagnostic contract. There are no dependency, lockfile,
workflow, auth, model, permission, or retry-policy changes.

## Verification

- Red before the production change: four TypeScript cases lost the
native reason or host stage. A Rust case returned `unclassified` for the
new closed categories.
- Final focused checks: 320 tests passed: 241 across sidecar
classification, Grok admission, the real ACPX client, recovery, runtime
adapter, and Copilot installation; all 79 host integration tests also
passed in isolation. These include nine startup-stage cases, translated
errors, actual missing Pi skill materialization, missing Copilot/Cursor
credentials, and a missing workspace binding.
- Review follow-up: a translated Copilot rejection lost its stage, and a
real Pi skill-loading failure reported configuration. Both regressions
failed before the two-line repair and pass afterward. Stage recording
now follows provider translation and starts before Pi skill acquisition.
The current unpublished Copilot 70 identity was regenerated.
- Formal review follow-up: missing Copilot and Cursor credentials
reported configuration. Both cases failed before the two-line stage
repair and pass afterward. A missing-workspace case also verifies the
separate binding catch.
- Seven Rust unit checks and 17 sidecar transport integration checks
passed. The pipe checks preserve known categories and reject unknown
suffixes without disclosing the provider message.
- `pnpm -r typecheck` and `pnpm build` passed. Rust formatting and the
generated profile/source identity checks passed.
- Host integration history is retained: one complete 73-case host run
passed. Other batches hit credential-port ownership errors in different
existing cases; the four new stage regressions passed. The final 79-case
host run passed after the last review fix. The prior intermittent lease
failures remain recorded and are not claimed fixed here.
- `pnpm test:run` was attempted and stopped after repeated embedded
Postgres bootstrap failures. The same local database failure predates
this change. Full CI must pass before merge.
- A metadata-only probe with pinned Grok 1.0.13, private HOME, synthetic
credentials, a local catalog, and no inference passed fresh Grok
4.7/High admission and cold restore. This is supporting ABI evidence. It
does not prove live subscription execution.
- Initial CI history is retained: server shard 1 failed its unchanged
final rate-limit slot test (two requests allowed instead of one). No
server policy code changed here. The review follow-up requires fresh
full CI; this is not claimed fixed.
- Second CI history is retained: chat shard 2 failed the unchanged Slack
session-sync quiet-queue test (three selects during the quiet window).
The first rate-limit failure did not recur there. No chat or server
policy code changed in this PR. Final head
654bc5aa21 passed all 61 contexts (56
success, four intentional skips, one informational neutral review). Both
final-head reviews are 5/5 with no new findings; the Paperclip app
publishes its completed review as an informational neutral check.
- Live native acceptance is still pending. After deployment, retry the
original bounded task and inspect its actual shell output, exit status,
run result, and saved task state. This PR fixes the evidence gap; it
does not claim the startup incident is resolved.

## Risks

- Shared startup failures can now receive a stage category. The private
weak map cannot import provider-defined fields. Known error codes keep
priority.
- A category could point at the wrong stage. Tests inject failures at
the installation, sandbox, credential, and runtime boundaries. Existing
cancellation and cleanup tests remain in place.
- The Copilot profile identity changes because shared source changes.
Historical identities remain decodable and cannot authorize a current
launch.
- The next live failure may require another functional repair. Admission
remains closed until the requested configuration is verified.

## Model Used

- OpenAI Codex, based on GPT-6, with reasoning, tool use, and code
execution. The exact deployment model ID and context-window size are not
exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (focused checks; full-suite
environment failures are recorded above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-27 20:53:02 -05:00
DottaandPaperclip 4e27a82a42 fix(server): wake approval queue seeding from committed changes (#17510)
## Thinking Path

> - Paperclip manages work and human decisions for AI-agent companies.
> - Automatic decision queues store membership for pending attention
sources.
> - The old attention maintenance scan created that membership as a side
effect.
> - Moving approval retention out of the scan also removed approval
queue discovery.
> - Queue API consumers do not build the attention feed first, so they
can miss pending PR approvals.
> - This pull request gives approval queue discovery its own worker and
commit signals.
> - The worker becomes quiet after it saves every missing membership.

## Linked Issues or Issue Description

Refs #17497. Stacked on #17508, following #17505.

This is a correctness dependency for the approval retention migration in
#17497. Do not treat that migration as complete without this
queue-population path. A real PostgreSQL regression found no PR queue
when a pending approval had a linked PR work product and the maintenance
feed excluded approvals.

Related: #13364 proposes pruning stale seeded memberships. This PR
preserves existing membership retention and does not implement pruning.
#15149 documents the decision-queue APIs.

## What Changed

- Add an approval queue worker that selects missing PR-queue memberships
from existing durable records. Bound each pass to 100 rows. Continue
past failed rows and revisit them on retry.
- Reuse the queue service's seed writer through a normalized source
contract. Keep queue rules, uniqueness, and audit writes in that
service. Keep feed readers compatible.
- Wake from committed approval and link changes, issue/company
visibility, PR work-product changes and imports, and queue-rule changes.
Reuse the existing transaction boundaries.
- Recheck approval status under a row lock. Serialize queue-rule changes
with the seed writer. Preserve required recovery during drain and
shutdown.
- Register startup reconstruction under the existing maintenance switch.
Leave other attention-source seeding with its current owner until its
own migration.
- Document the complete producer audit and stack dependency. Add a
validated external PostgreSQL option to the existing queue route suite
so it runs on hosts without embedded database support.

## Verification

Current tested revision: `7948136ef8`. This PR targets master after
#17508 merged as `8406047b4e`. Normal ancestry reconciliation preserved
the complete tested integration tree, before the review repairs below.
Only the original 25 seeding paths differ from master, and database
schema/migration metadata is identical.

- **56 cases pass in the three complete affected suites:** 28 approval
seeding, 6 queue routes and 22 task-watchdog cases. Server typecheck and
production build pass.
- Three real PostgreSQL retry regressions failed then passed: new
matching arrivals cannot postpone a recovered failure indefinitely;
successful partial progress resets old failure penalties; an empty
cursor rollover does not reset outage backoff. Exact one- and two-second
retry assertions, eventual complete recovery and existing
capped-backoff/company isolation tests pass.
- Three genuinely empty external databases failed before setup repair
because tables were missing. Each suite now applies migrations itself.
All 55 cases passed on a second set of fresh databases; adding the final
rollover guard then passed all 56 cases using those now-migrated
fixtures. No production migration or table is added.
- The initial expanded fresh run passed 54 cases but failed a cleanup
expectation that omitted a required cursor rollover pass. The fixture
now permits that explicit continuation while preserving exact deadline
and complete-membership assertions. All failure records remain in
`doc/idle-approval-queue-seeding.md`.
- The prior integrated revision `4e725e2055` passed **522 cases in
fifteen complete producer/consumer suites**, plus types/build. Its
unchanged consumer code and historical deadlock, cross-company
amplification, backoff, gateway integration and setup evidence remain
documented; those counts are not relabeled as current acceptance.

Commands: server Vitest for approval queue seeding, queue routes and
task watchdogs; server typecheck/build. Fresh exact-head
Greptile/Ponytail reviews and all CI checks remain required. No full
local repository-suite, deployed suspension or cost claim is made.

## Risks

- Signals cover the supported in-process service writers. Independent
SQL writers and multiple server replicas need a separate notification
contract.
- The source predicate must stay aligned with unscoped attention
maintenance. The first linked issue, company scope, pending status, and
execution visibility are tested. A pending approval itself keeps its
linked external conversation active.
- Existing enabled seed rules can re-add manually removed matching
memberships. This preserves the old scan's behavior. Completed or
archived membership is not pruned.
- Queue seeding and retention are separate owners. Other attention
sources still need their own event-driven discovery before the aggregate
scan can be retired.
- This PR does not extend Cloud wake coverage, relax sleep admission, or
claim measured host/database suspension.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact
serving model identifier and context window were not exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-27 20:33:01 -05:00
DottaandPaperclip 8406047b4e fix(server): schedule join request retention from durable deadlines (#17508)
## Thinking Path

> - Paperclip manages agent companies and their access requests.
> - Pending join requests appear in the attention feed and can age into
automatic archival.
> - The maintenance tick reads those requests even when no archive is
due.
> - Existing request rows already store the activity timestamp needed
for retention.
> - Committed changes can wake a worker that registers the next actual
deadline.
> - This PR gives join-request retention its own bounded worker and
removes its recurring feed scan.
> - Idle and resolved history stop producing retention traffic without
changing access approval.

## Linked Issues or Issue Description

**What happened?**

Every retention maintenance pass reads pending join requests through the
attention feed. Empty, resolved, kept, and future-due states still
participate in recurring discovery.

**Expected behavior**

Committed request and retention changes wake the owner. It waits until
the next actual archive deadline and stops querying when no eligible
request remains. Existing approval and visibility behavior is preserved.

**Steps to reproduce**

Start an idle company with no pending join requests, or with requests
younger than the 90-day archive threshold. Observe the maintenance feed
repeatedly select join_requests. No state change is needed to cause
those reads.

This PR targets master after #17505 merged. Approval retention is
#17497. Other attention source kinds remain separate migration duties.
No duplicate open join-retention PR was found.

## What Changed

- Add a join-request retention owner that preserves pending status,
active-company visibility, matching invite ownership, and keep/archive
controls. Invite expiry or revocation retains the existing feed policy.
- Reuse one `createDecisionRetentionSweep` for approval and join paging,
atomic retention and finite scan cycles that revisit failures despite
continuing arrivals. Eligibility and source-row locks stay with each
owner.
- Register real saved deadlines, rounded upward in SQL to millisecond
precision, with the existing coordinator. Preserve 100-row pages and
thirty-second notification coalescing without idle polling.
- Materialize unseen join state inside the existing authorized
keep/archive/revive transaction, preserving archive choices and source
locking.
- Notify before request inserts and updates in the existing acceptance
and Slack transactions. Use a small SQL-only helper for replay and
approval/rejection updates. Company status, budget pause/resume,
deletion, and retention controls notify this owner too.
- Preserve the approved-only API-key consumption path without a new
wrapper; it cannot produce pending retention work.
- Exclude join requests from maintenance feed collection while retaining
normal UI reads.
- Add an idempotent concurrent partial pending index and prune old
schema snapshots through the existing generator.
- Document producer coverage, boundaries, failure evidence, and
remaining work.

## Verification

Test revision: `df1815fbba`. Normal integration of landed workspace
precision `d23dd19750` has an identical complete source tree. This
remains only the join-retention duty and its shared approval-retention
mechanics, targeting master.

- **55 worker cases pass**, including two new PostgreSQL regressions
that failed before repair: continuing overdue arrivals no longer
postpone a recovered failure indefinitely, while an entire failed page
still allows later healthy rows to progress.
- The nine complete suites passed 341 cases but failed an attention
cleanup hook after its 16,400-run case. After vacuum/analyze of only
that disposable fixture, the full 31-case attention suite passes
unchanged. Together with the original eight passing suites (311 cases),
all 342 distinct cases pass across complete-suite executions. Recovery
is established; the original cleanup slowdown is not claimed explained.
Server typecheck and production build pass.
- Earlier CI at `df1d4d0` failed the unchanged decision-expiry restart
deadline assertion (a thirty-second retry appeared before the saved
expiry). All twenty unchanged expiry cases pass locally, but the CI
timing cause remains unproven. No retry, timeout relaxation or assertion
change is used to dismiss it; fresh final-head CI remains required.

- **Five regressions failed before repair:** three unseen join controls
returned 404, and both owners advertised a microsecond deadline one
millisecond early. All pass. Two old real-producer expectations failed
after the correct rounding change; they now derive exact expected
milliseconds independently from database integer microseconds instead of
truncating a JavaScript Date.
- The initial timing-only concurrency test still passed a missing-lock
mutation. It now observes the actual PostgreSQL wait before releasing
the writer. Production passes; removing the lock fails with zero waiters
instead of one. Source is restored before commit. The archive-first
committed-order case confirms that later activity stays archived until
explicit revival; no broader race guarantee is claimed.
- Generated concurrent/idempotent `0393_open_grandmaster.sql` adds only
the pending join index. All 391 parent journal entries and historical
SQL files remain unchanged. Five generated snapshots remain (0389–0393).
Migration safety/numbering, schema drift and actual apply/replay on six
isolated PostgreSQL fixtures pass.
- Existing lifecycle cases retain real HTTP invite
acceptance/approval/rejection, Slack identity linking, rollback/outer
commit, source visibility, failure rotation, drain/resume and shutdown
coverage. Earlier stack setup and CI failures remain in the owning audit
and stack ledger; they are historical evidence, not this head's
acceptance.

Commands: server Vitest for `join-request-retention-work`,
`approval-retention-work`, `approvals-service`, `attention-service`,
`decision-retention-service`, `budgets-service`, `tool-gateway-service`,
`server-startup-feedback-export`, `scoped-deadline-work`; server
typecheck/build; DB migration checks, snapshot drift and actual
apply/replay. Fresh exact-head CI, security and Greptile/Ponytail 5/5
remain required. No full local repository-suite, deployed suspension or
cost claim is made.

## Risks

- A future supported producer must signal its transaction. Independent
SQL writers and another active server process are outside the
process-local notification contract.
- A writer that wins the source lock first is rechecked before archival.
If archival commits first, later activity preserves the archived state
until explicit revival. Existing invite replay expires before the
retention cutoff and Slack replay deduplicates. Future producers needing
renewed visibility must request revival. Cross-table visibility retains
the existing contract.
- The worker registers only under the existing maintenance enablement
switch. The internal deadline does not add Cloud alarm coverage or relax
conservative sleep blockers.
- New eligibility can wait within the thirty-second aggregate
notification window; known deadlines preempt it.
- Archival preserves pending approval status. Other attention sources
and aggregate maintenance remain separate work.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact
serving model identifier and context window were not exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
canary/v2026.1028.0-canary.2
2026-10-27 20:03:16 -05:00
DottaandPaperclip d23dd19750 fix(server): preserve terminal workspace cursors and retry deadlines (#17505)
## Thinking Path

> - Paperclip manages agent work and its execution workspaces.
> - Workspace recovery scans candidates and retains deadlines for
unfinished work.
> - Rounded database timestamps can prevent a scan from advancing.
> - Separate clock reads can give one retry obligation different
deadlines.
> - Repeated inspection must preserve both scan progress and the saved
retry time.
> - This PR keeps full cursor precision and computes each retry deadline
once.
> - Recovery can finish its scan and wait until work is actually due.

## Linked Issues or Issue Description

**What happened?**

Terminal workspace recovery can repeatedly inspect the oldest skipped
candidate when its timestamp contains microseconds. Eligible workspaces
behind it never reach cleanup. Separately, nested-clone migration and
isolated-repository rehome recovery use separate clock reads for their
retry gate and advertised deadline. A later inspection can move the
advertised deadline backward.

**Expected behavior**

Each bounded scan advances using the complete timestamp and ID. The
retry gate and advertised deadline represent the same saved time,
including when the clock advances between calls.

**Steps to reproduce**

For cursor starvation, place two skipped workspaces with microsecond
timestamps before an eligible terminal workspace and use a one-row page.
For deadline drift, inject a clock that advances one millisecond per
read, run disabled migration or rehome recovery, then inspect it again
before the retry time.

Stacked on #17497. Related cursor fixes #17479 and #17482 have separate
owners. This PR owns workspace recovery cursor and retry precision.

## What Changed

- Preserve the database timestamp as text in the private timestamp/ID
cursor.
- Compute one retry time per migration or rehome outcome and reuse it in
the existing maps.
- Test the cursor with real PostgreSQL microsecond timestamps and the
retry boundary with a deterministic advancing clock.
- Support validated disposable external test databases in the affected
fixtures, retaining embedded fallback.
- Document the evidence in `doc/idle-terminal-workspace-pagination.md`.

## Verification

Current test revision: `9d846c84e76c90893ab62a09548c11bbdb627771`,
followed by docs-only head `a34dc50de3`, which received both 5/5 reviews
and passing CI. This candidate normally integrates landed approval
retention `553a004926`. Every runtime and acceptance-test source matches
that tested/reviewed tree; only four independent Runner E2E accounting
files and the ledger differ. Migration metadata is unchanged. Existing
local checks are reused for their unchanged source; fresh final-head CI
and reviews are required against master. This remains the
terminal-precision duty.

- **136 cases pass** in four complete suites: execution workspace
service, terminal worker, isolated repository rehome and nested-clone
migration. Server typecheck and production build pass with the current
Runner vendor.
- The preceding `f0f0de1f6f` run also passed all 136 on three newly
created, unmigrated PostgreSQL databases. The suites themselves applied
migrations. The current run reuses those clean fixtures after
integrating the parent's approval wake repair; it is not presented as a
fresh-database run.
- The regressions retain real PostgreSQL microsecond timestamps and an
advancing clock. Before repair, skipped rows starved eligible cleanup
and early inspection moved a saved retry by one millisecond. Current
assertions preserve exact progress/deadline behavior; no timeout or
business policy is weakened.
- Earlier external-fixture setup tests failed with missing
company/restore-point schema. Their setup repair and red/green record
remain in `doc/idle-terminal-workspace-pagination.md`. This PR adds no
migration or new queue.
- Earlier stack integration receipts and failures remain in the stack
ledger and `doc/idle-approved-tool-recovery.md`; they are historical
parent evidence, not this head's acceptance. No full local repository
suite, deployed suspension, fleet cost or live provider qualification is
claimed.

Commands: server Vitest for `execution-workspaces-service`,
`terminal-workspace-work`, `isolated-repository-rehome` and
`nested-clone-migration`; server typecheck/build. Fresh exact-head CI
and Greptile/Ponytail 5/5 remain required before merge in dependency
order.

## Risks

The cursor remains private query state. Retry intervals, eligibility,
cleanup authority, and shutdown behavior are unchanged. No schema,
queue, external alarm, or additional polling loop is introduced. The
tests establish workspace recovery behavior, not host suspension or cost
savings.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact
serving model identifier and context window were not exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
canary/v2026.1028.0-canary.1
2026-10-27 19:42:58 -05:00
DottaandPaperclip 553a004926 fix(server): schedule approval retention from durable deadlines (#17497)
## Thinking Path

> - Paperclip manages agent work and should become idle when it owes no
background work.
> - The attention feed archives old pending approvals under the existing
90-day policy.
> - The server rebuilds the feed on each maintenance tick, even when
approval state has not changed.
> - Some approvals have never been read, so retention rows alone cannot
define the work.
> - Approval records and committed eligibility changes define the real
deadline and wake triggers.
> - This PR gives approval retention its own deadline worker and reuses
the existing retention and archive-delivery contracts.
> - The approval lane stops idle scans while other attention sources
remain separate migration work.

## Linked Issues or Issue Description

Stacked on #17490. This PR owns approval retention only. Archive
notification delivery is already separated in #17475. Other
attention-source retention and remaining recurring server duties remain
in the stack inventory. The related-PR search found no overlapping
approval-retention worker migration.

**What existing behavior does this improve?**
Automatic archival of old pending approvals from attention.

**Current behavior**
The shared maintenance tick builds the attention feed for each active
company. That query discovers pending approvals and materializes
retention records before checking their age. An unchanged or empty
approval source is read repeatedly.

**Proposed behavior**
Reconstruct eligible approvals at startup. Schedule the earliest
approval activity timestamp plus 90 days. Wake after committed changes
to approvals, links, issue visibility, company activity, or retention
controls. Stop scheduling when no eligible approval remains. Archive
records through the existing version checks, audit writes, and
notification outbox.

**Reason and benefit**
Idle approval retention should generate no recurring SQL. Unseen
approvals still need eventual archival. A stale candidate must not
archive an approval that was resubmitted while the worker waited for its
source lock.

**Breaking changes**
None intended. The archive age, eligibility, keep controls, audit, and
notification policy remain. Normal attention reads still include
approvals. No Cloud sleep blocker is relaxed.

## What Changed

- Restrict tool-review signals to their existing terminal transition
owners (approve/reject/cancel). Creation/link/revision/resubmit signal
retention only.
- Coalesce retention notifications under one thirty-second aggregate
read budget. Known deadlines still preempt it; no notification or
durable deadline means no scan.
- Add a bounded approval-retention worker. It uses the shared deadline
coordinator and reads pending approvals directly.
- Share first-linked-issue ordering and execution visibility through
`approval-visibility.ts`, used by both the feed and worker. Preserve
actor-specific read filtering and locked archival rechecks.
- Materialize unseen approvals on demand inside manual
keep/archive/revive transactions. Use the saved activity timestamp and
existing versioned state; retain company scoping and rollback on
failure.
- Add awaited pre-write signals to approval, budget, tool, link,
visibility, company, and retention mutation owners. Preserve outer
transaction ownership; keep provider calls outside SQL wrappers.
- Narrow retention inputs to the source fields the existing service
uses. Signal materialization only for inserted or changed state so
unchanged reads converge.
- Exclude approval collection from the maintenance feed before its
query. Other source kinds remain in that sweep.
- Add the pending-only approval deadline index with concurrent,
idempotent creation. Keep the five newest migration snapshots.
- Add real-Postgres lifecycle tests and startup wiring tests. Enable
validated external test databases for existing attention and budget
suites.
- Document producer coverage, quiet conditions, recovery and remaining
scope in `doc/idle-approval-retention.md` and the stack ledger.

## Verification

Current test revision: `dd844f4e8cbbf1db681a774dc31cd552301bef48`,
followed by docs-only `3efda33812`, which received both 5/5 reviews and
passing CI. This candidate normally integrates landed Grok `2adcde162e`.
All runtime and acceptance-test sources are byte-identical to that
tested/reviewed tree; only four independent Runner E2E accounting files
and the ledger differ. Migration 0392 and the parent journal prefix are
unchanged. Local checks are reused for their unchanged source; fresh
final-head CI and reviews are required against master. This remains the
approval-retention duty.

- **315 cases pass** across eight complete suites: approval
retention/service, attention, decision retention, budgets, tool gateway,
server startup and scoped deadlines. All 30 real-PostgreSQL lifecycle
cases pass. Server production typecheck and build pass.
- **Four regressions failed before repair**: unseen young
keep/archive/revive returned 404; the manual keep rollback case could
not reach its update. They now pass. A failed keep update leaves no
retention row or audit; retry succeeds with one audit.
- Both linked-issue orders are checked against the feed: hidden
first/visible later is excluded; visible first/hidden later remains
eligible. The feed and worker use the same owner query, with actor read
filtering preserved.
- Generated concurrent/idempotent migration
`0392_small_frightful_four.sql` adds only the pending approval index.
Every historical parent SQL migration and all 390 parent journal entries
are unchanged. Five generated snapshots remain (0388–0392). Schema
drift, migration safety/numbering and actual fresh apply/replay on five
isolated PostgreSQL databases pass.
- Existing precision, transaction, drain/resume, failure fairness and
shutdown tests remain. Earlier transaction-stub, budget-import,
diagnostic-client and CI failures are recorded in the owning audit and
stack ledger; later passes do not erase them.

- Three additional real regressions failed before the wake repair:
ordinary mutations emitted tool-review signals in both resolution cases,
and sequential tenant writes caused repeated global reads. Tests now
prove terminal approve/reject/cancel signals and one aggregate
reconciliation per thirty-second window with a different company's
pending approval retained. The aggregate budget is stricter than a
separate per-company budget and matches the old default cadence; known
deadlines preempt it.
- The initial coalescing fixture expected the concrete pending-work hold
to clear before reconciliation; it now preserves that hold and measures
SQL. The first broad repair passed 313 cases but failed the existing
parked-review cancel case. The original cancel producer is restored; all
315 now pass without weakening assertions. At old head `5991f257bb`,
security was action-required and Greptile returned service error
`8fb1835c-e564-476b-b802-d5be164ea915`. Fresh repaired-head results are
required, not assumed.

Commands: server Vitest for `approval-retention-work`,
`approvals-service`, `attention-service`, `decision-retention-service`,
`budgets-service`, `tool-gateway-service`,
`server-startup-feedback-export`, `scoped-deadline-work`; server
typecheck/build; DB snapshot drift, migration checks and fresh
apply/replay. Fresh exact-head Greptile/Ponytail and CI remain merge
gates. No full local repository suite or deployed host suspension/cost
claim is made.

Earlier stack verification below is historical evidence at the
explicitly named revisions, not this head's acceptance:

Approved delegated Test-tab calls now evaluate caller and target
policies in the existing claim transaction. Real HTTP regressions first
showed a missing caller charge and shared-limit self-rejection. All 78
connection-management cases and 92 gateway cases now pass on the owning
branch, along with server production typechecking. All 78
connection-management cases also pass at combined tip
`d25f830db89c1623bff3547bfc68015c1c66e0cf`. The three new cases cover
shared charges, separate charges, rollback on denial, duplicate
approval, and rejection of a later authorized call. Intermediate
counter/identity fixture failures and a reused-database Slack collision
are recorded in [the owning
audit](https://github.com/paperclipai/paperclip/blob/20557d2c3dd0949c734eaa8095fd78b9b07d7fd5/doc/idle-approved-tool-recovery.md#delegated-test-tab-admission-after-policy-changes).
The earlier combined checks below remain evidence for their stated
revisions. Current-head CI and review remain required.

Earlier published head: `20557d2c3dd0949c734eaa8095fd78b9b07d7fd5`. This
remains a single-duty PR stacked on #17490.

- Integrated pinned master `8548173fc9efdc0fa07644fb92dcc0b6851a95e5`
with normal merges. Every branch passed generated-schema checks and its
target-branch migration-order gate. Existing SQL is unchanged; the 25
stack migrations follow master’s new migration.
- Duty-specific producer coverage, lifecycle tests, and earlier failure
records:
[2026-10-24-recurring-server-sleep-stack.md](https://github.com/paperclipai/paperclip/blob/76a49810c7ff403ae7720bef0d453747bd844731/doc/plans/2026-10-24-recurring-server-sleep-stack.md).
- Combined tip `51e6cbe67979b55b340ea9f0f434c5b5c682ccd6`: **355
startup/cancellation/recovery/dispatch/preservation tests**, **75
connection-management tests**, and production server typechecking pass.
These combined checks do not mean every intermediate PR independently
ran the full repository suite.
- Real master-to-stack database upgrade passes: **379 unchanged prior
migration receipts plus 25 new migrations**, with the gateway
responsibility fixture preserved.
- Owning-branch checks: 84 runner installation/settlement cases plus
runner typechecking and generated TypeScript/Rust profile agreement; 253
GitHub review workflow cases. Stop-attribution and held-worker readiness
regressions failed before their fixes.
- Failure accounting: the initial combined run passed 349 cases and
failed six idle assertions in a reused preservation database containing
old cancelled runs. Repeating the combined command with a fresh
preservation database passed. Earlier invalid project selection, merge
import/type errors, and the interrupted full local suite are not counted
as passing checks.
- **Latest-head CI and review remain required.** Prior CI findings
include GitHub idle observation, route teardown, attention cleanup,
touch-picker E2E, workspace health, and voice-cleanup timing; local
passes do not erase those failures. No full local repository-suite,
deployed sleep, fleet cost, or live Copilot qualification claim is made.

## Risks

- Signals cover the application database owner and its child pools.
Independent SQL writers and other active processes need an explicit wake
integration. Startup reconstructs committed durable state.
- The source row lock prevents stale approval-activity archival. It does
not add cross-table serializability for concurrent link, visibility, or
company-policy changes.
- Newly changed eligibility can wait up to the thirty-second
notification window, matching the former default maintenance cadence.
Saved deadlines still preempt; no new idle polling is added.
- Failed records retain recovery debt. Bounded cursor progress lets
later rows proceed without dropping failures.
- The shared retention transaction now groups materialization and
activity-version updates. Existing attention and retention consumers are
tested against PostgreSQL.
- Kept, archived, resolved or inactive-company approvals can become
quiet, but their supported mutations must retain their wake
notifications.
- The local scheduler cannot wake stopped compute. This PR neither adds
approval deadlines to Cloud's external alarm contract nor claims
measured host or database suspension.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact
serving model identifier and context window were not exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-27 19:23:24 -05:00
DottaandPaperclip 2adcde162e fix(server): wake Grok dispatch from committed reservations (#17490)
## Thinking Path

> - Paperclip manages agent work and should become idle when no work is
owed.
> - Grok routine dispatch already persists each outbound intent before
delivery.
> - Its dispatcher starts a one-second timer whenever public MCP is
configured.
> - That timer queries settings even when Grok is disabled, and scans
dispatches when enabled but idle.
> - The reservation transaction and feature-setting transaction already
define when work can become available.
> - This PR connects those writes to the shared coordinator and retains
the Grok service's claim and delivery policy.
> - The worker becomes quiet without adding a queue table or replaying
uncertain sends.

## Linked Issues or Issue Description

Stacked on #17486. This PR owns only Grok routine dispatch. It does not
include decision retention or other remaining maintenance owners.
Related open Grok PR #17278 concerns Runner credential-lease fixtures,
not this dispatcher. The duplicate search found no overlapping
dispatcher migration.

**What existing behavior does this improve?**
Grok routine dispatch recovery in the server.

**Current behavior**
With public MCP configured, the dispatcher reads instance settings every
second, including when Grok is disabled. With Grok enabled, it scans for
prepared dispatches every second. The baseline regression observed three
queries in 3.2 seconds with Grok disabled.

**Proposed behavior**
Reconcile once at startup. Wake from committed dispatch reservations and
experimental-setting changes. Schedule another pass only while prepared
dispatches remain or reconciliation failed. Disabled and settled workers
perform no recurring SQL or provider calls.

**Reason and benefit**
Remove unnecessary tenant database activity while preserving dispatch
recovery, current authorization, provider outcome semantics, and
shutdown ownership. This follows the existing sleep-aware maintenance
direction; ROADMAP review found no competing dispatcher redesign.

**Breaking changes**
No public API or provider payload change. The internal dispatcher start
method receives the shared coordinator; that coordinator now owns
shutdown. Sending, accepted, unknown, and terminal dispatches remain
non-replayable. This does not extend Cloud wake coverage.

## What Changed

- Register the Grok worker with the existing coordinator; remove its
independent timer.
- Signal reservation and experimental-setting writes before commit.
Nested writes follow their outer transaction.
- Preserve ten-row batches, durable send claims, current authority
checks, and existing cancellation/uncertainty policy.
- Join admitted work during shutdown and leave later rows for restart.
- Add a concurrent, replay-safe partial index for prepared dispatch
lookup. Keep only the five newest generated snapshots.
- Reuse the existing real OAuth/agent/routine fixture for lifecycle
tests and add a validated external PostgreSQL test option.
- Document producers, recovery, quiet conditions, and remaining
host-sleep limitations.

## Verification

Test revision: `6d81746fe260ee879442e7e0c879cfe88c250c8e`, followed by
docs-only head `679f523d78`, which received both 5/5 reviews and passing
CI. This candidate normally integrates landed target `8dfed893d8`. Its
entire runtime and acceptance-test sources match that reviewed/tested
tree; only four independently merged Runner E2E accounting files and the
ledger differ, preserved exactly. Migration 0391 and the parent journal
prefix are unchanged. Local checks were not repeated on unchanged
runtime; fresh final-head CI and reviews are required.

- **273 cases pass** in nine complete suites: real-PostgreSQL dispatch
lifecycle, Grok qualification, server startup, scoped deadlines and all
five instance-settings suites. Server typecheck and production build
pass with the updated Runner vendor.
- The baseline disabled-feature regression failed with three SQL reads.
Existing lifecycle cases cover first reservation, restart,
non-replayable history, re-enable, outer commit/rollback, drain,
failure/retry, concurrent enqueue, paging, foreground send race, expiry,
settings commit/rollback, and shutdown.
- Generated migration `0391_third_roulette.sql` adds only the
concurrent/idempotent prepared-dispatch index. All 389 parent journal
entries are unchanged; five generated snapshots remain (0387–0391).
Migration safety/numbering, schema drift and actual fresh apply/replay
on two isolated PostgreSQL databases pass.
- The earlier 259-case selection and 40,000-row generic-plan probe
remain historical evidence: both unchanged queries used the partial
index, six shared buffers including the routine join, 0.032ms / 0.025ms
local execution. No new deployed suspension or cost claim is made.
- Earlier browser CI failed during agent-chat fixture restoration (lock
timeout/HTTP 500), followed by a test seeing the experiment left
enabled. Those failures remain recorded in the owning audit; they were
not dispatcher assertions. Later pre-reconciliation `a74305e2b4` checks
were terminal and green. Fresh CI and both 5/5 bot reviews are required
for this head.

Commands: server Vitest for `grok-dispatch-work`, `grok-bot-routine`,
`server-startup-feedback-export`, `scoped-deadline-work` and the five
`instance-settings-*` suites; server typecheck/build; DB migration
safety/numbering, snapshot drift and fresh migration apply/replay.

## Risks

- Signals are local to the shared database owner and its child pools.
Independent SQL writers and other replicas need explicit wake
integration. Startup recovers committed prepared rows.
- Disabled Grok retains prepared rows and resumes after a committed
settings change. This does not relax Cloud's binding/assignment/hold
guards.
- Existing send-error cancellation policy is preserved. Sending or
unknown outcomes are not reset or automatically replayed; this is not an
exactly-once delivery change.
- Shutdown awaits the current send within existing transport timeouts
and stops before the next row.
- The index adds write/storage cost but bounds queries independently of
completed history. Existing migration execution runs concurrent index
creation outside a transaction.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact
serving model identifier and context window were not exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-27 19:07:53 -05:00
DottaandPaperclip 8dfed893d8 fix(server): wake decision target checks from committed changes (#17486)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Decisions can depend on strict references to other tasks.
> - A cancelled, deleted, or completed target can make a decision
obsolete.
> - The server currently scans open decisions on every maintenance tick
to find those changes.
> - The owning issue writes already have transaction boundaries that can
signal the change.
> - This pull request reconciles affected decisions after commit and
reconstructs open targets at startup.
> - Valid open decisions and completed history stop causing periodic
SQL.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Decision target invalidation. This is slice 12 of the recurring-server
sleep stack, based on #17482. TTL expiry is #17479 and continuation
delivery is #17478. Retention remains separate work.

**Subsystem affected**
Server decision policy, issue mutation and deletion, and startup wiring.

**Current behavior**
The aggregate sweep repeatedly reads open decisions and their strict
targets. It shares a maintenance tick with unrelated jobs even when
every target remains valid.

**Proposed behavior**
Committed decision creation and terminal issue updates request a scoped
check. Deletion requests a bounded full scan because its target links
can cascade away. Startup reconstructs open target state. Once
reconciliation settles, no target scan remains scheduled.

**Reason and benefit**
Remove normal empty polling without losing target invalidation,
transactional delivery, or restart recovery. Reuse existing decision
rows and target links instead of adding a table.

**Breaking changes**
No API or schema change. The main tick stops calling the combined
decision sweep after its four duties have separate owners. The public
combined method retains its default behavior.

**Additional context**
The initial decision implementation stored secondary links as well as
primary links. This migration does not assume a new legacy backfill.
Independent SQL writers still require an explicit signal integration.

## What Changed

- Replace the periodic target scan with committed-change activation and
bounded startup reconstruction. Keep the existing decision policy and
expiry/activity owner.
- Require production notifications to identify an issue or decision.
Before deletion, lock the issue and capture its linked open decisions in
the same company; emit their IDs before links cascade. This avoids
resetting full scans and fences concurrent proposal creation.
- Reuse that owner helper for issue removal and routine failure cleanup.
No linked decisions means no target signal. Remove company deletion's
ineffective global signal; its pre-existing restrictive decision foreign
keys remain unchanged.
- Retain PostgreSQL timestamp text in paging cursors so rows within one
millisecond cannot repeat indefinitely.
- Remove the unused combined internal decision sweep after its final
duty is split; exercise each owner directly in the existing behavioral
tests.
- Keep query policy and notification discovery in `decision-targets.ts`.
The coordinator registration contains lifecycle wiring only. Add no
queue table, per-route recovery wrapper or external-wake exemption.

## Verification

Current candidate normally integrates landed effect recovery
`e9e84bb8a1`. Its runtime and acceptance tests are byte-identical to
tested target head `85ce6545bf`; the only new source files are four
independently merged Runner E2E accounting files, preserved exactly. The
248-case/type/build evidence comes from test revision
`54936a919f36c2ea421e7ce8e15912b1105b8dcc`, followed by
documentation-only changes. Fresh current-head CI and reviews are
required. This remains the target-invalidation duty only.

- **248 cases pass** in all seven complete suites: 25 target, 80
routine, 54 decision, 26 effect-recovery, 20 expiry, 42 startup and one
scoped-deadline case. Database suites use separate migrated PostgreSQL
fixtures. Server production typecheck and build pass.
- **Four regressions failed before repair:** paging across microsecond
timestamps, deletion behind a full valid startup page, unnecessary
global work from unrelated deletion, and unnecessary work after deleting
an empty company. All four pass.
- A real concurrent proposal is observed waiting on the issue-row lock
between pre-delete discovery and deletion commit. Removing that lock
made the test fail because the proposal committed instead of waiting.
Restoring the implementation passes. The mutation was restored before
commit.
- Existing real producer tests cover commit/rollback, deletion with
scoped hints in one outer transaction, secondary targets, re-opened
tasks, valid/lenient history quietness, startup paging, failure retry,
drain/resume and shutdown. The routine failure case now requires no
signal when its cleanup has no linked decision.
- The earlier dropped-hint and drain-fixture failures remain in [the
behavior audit](doc/idle-decision-targets.md). Extra upstream
embedded-only tests attempted on the parent could not initialize
locally; CI must cover that environment. No new schema migration or
table is added.

The final review cleanup removes the combined internal `sweepExpired`
method, cursor, flags and unused target-policy export. A
repository/package/SDK audit found no remaining production caller or
public exposure. Behavioral tests now call the dedicated workers;
isolation tests prove TTL does not recover stale effects and effect
recovery does not expire open decisions. HTTP and plugin SDK contracts
are unchanged. The full 248-case selection passes after this cleanup and
parent reconciliation; types/build pass with the updated Runner vendor.


Current-head CI run 40602159553 (attempt 1) failed in the workspace CLI
group: company import/export startup exceeded its 30-second health
deadline, and the dense ZIP case exceeded its unchanged five-second
timeout. The preceding UI group passed all 4,169 cases. Both CLI files
and the transfer implementation are unchanged from the landed parent.
The complete ZIP-transfer file passes all 17 cases locally on this exact
head; the CLI startup E2E is skipped because the host's embedded
PostgreSQL bootstrap fails. This does not prove Linux startup locally.
The failure logs remain recorded; no timeout or assertion is changed.
One bounded retry of the failed jobs is requested after the workflow
completed. Its result and the current-head gates remain required before
merge.

Commands: server typecheck/build and Vitest for `decision-target-work`,
`routines-service`, `decisions-service`,
`decision-effect-recovery-work`, `decision-expiry-work`,
`server-startup-feedback-export` and `scoped-deadline-work`. Local
targeted acceptance does not establish deployed suspension or fleet
savings. Fresh exact-head CI and Greptile/Ponytail remain merge gates.

## Risks

- Deletion now holds the issue row lock while discovering dependent
decisions and completing the existing removal transaction. This is
needed to serialize new FK references. Provider calls are not placed
inside that transaction.
- Company deletion with decision-origin references still has the
pre-existing restrictive FK limitation. Removing the futile notification
does not implement a new company-data cleanup policy.
- Supported writers share the application DB owner and maintain existing
target links. Independent SQL writers, damaged links or multiple active
processes need their own wake contract; startup still reconstructs all
open targets.
- Cloud sleep blockers and durable external alarms are unchanged. Worker
quiet is not proof of deployed host/database suspension.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact
serving model identifier and context window were not exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
canary/v2026.1028.0-canary.0
2026-10-27 18:53:50 -05:00