Jannik/PaperClipAI
1
0 Fork 0
mirror of https://github.com/paperclipai/paperclip.git synced 2026-11-01 15:35:12 +01:00
Code Issues Packages Projects Releases Wiki Activity
5,851 Commits 1,351 Branches 2,207 Tags
0791da0cc5458c11da9d10f000645036ffd6f4e2
Commit Graph
5 Commits
Author SHA1 Message Date
DottaandPaperclip 96727b3cfe fix(server): schedule plugin jobs from saved deadlines (#17518) 
## Thinking Path

> - Paperclip manages agents and plugin work.
> - Plugins can declare cron jobs with saved deadlines.
> - The existing scheduler queries for due jobs every 30 seconds, even
when none exist.
> - Empty queries keep an otherwise idle database active.
> - This PR uses the shared scheduler and committed job changes to
discover work.
> - Existing job and run rows preserve deadlines and execution history.

## Linked Issues or Issue Description

**What happened?**
The plugin scheduler on the base of this stack runs empty periodic SQL.
Its registration path also advances overdue timestamps. An empty
replacement manifest can leave removed schedules active.

**Expected behavior**
No recurring SQL when no eligible jobs exist. Configured jobs wait for
their saved deadline. Changes reactivate scheduling after commit. Work
remains protected through drain and shutdown.

**Steps to reproduce**
Start the plugin scheduler with no jobs. After startup, observe queries
during a three-minute idle window. The new PostgreSQL regression failed
on the baseline and passes with this change.

**Paperclip version or commit**
Stacked on #17510 at `4d15c08428`. Deployment: source checkout, local
PostgreSQL, controlled plugin RPC. Related: #7905 covers empty-manifest
sync; this migration requires that behavior on both activation paths.
#11571 proposes company-scoped jobs; this PR preserves the existing
instance-scoped job contract.

## What Changed

- Install the concurrency test barrier before due job notifications can
dispatch.

- Make dispatch the sole reservation owner and preserve retained or
concurrently committed results when unregistering a plugin.

- Replace the private interval with the existing shared deadline worker.
- Signal job definition and run changes inside their owning
transactions. No new tables.
- Wake from plugin registration and worker status events. Retain
explicit scheduler wakes received during drain.
- Preserve overdue work and initialize only missing or changed
schedules.
- Recheck claims under a job-row lock. Keep plugin calls outside
transactions and refill capacity on completion.
- Commit completion and schedule advancement together. Retry failed
persistence without repeating the plugin side effect.
- Track admitted calls through shutdown and synchronize empty manifests.

## Verification

This PR now targets master after #17510 merged as
`4e27a82a4265c01f3ab4f05aaecc010a0026de75`. The normal landed-parent
merge preserves the entire verified `df78ca3b5e` tree byte for byte;
only final publication docs change afterwards. The PR still differs in
its original seventeen paths and has no database schema or migration
change.

Primary test revision: `36be0bd73363680d7666d94ff5f1dd28b27f4235`, on
parent `ba80060407`. Final integration of parent `51668b8240` preserves
all plugin/runtime/shutdown sources from that proof and passes all 63
cases in complete plugin, approval-seeding and scoped-deadline suites at
`56ce7c1d2a`, plus server types/build. Only six parent seeding/doc files
changed; this is not a rerun of all eleven suites. Only the original
seventeen plugin-scheduling paths differ from that parent; all other
tracked files, including migration history, match it.

- **247 cases pass in eleven complete suites:** plugin scheduler,
shutdown, warm standby, plugin environment-driver readiness, delivery
coordinator, both retention workers, approval seeding, decision expiry,
scoped deadlines and startup. Server typecheck and production build
pass.
- The new real PostgreSQL lock test failed before the repair: shutdown
remained pending while the completion update waited for a held run row.
After repair, shutdown reports the statement timeout while the lock
remains held; retry after release commits the saved result with exactly
one plugin invocation. All 34 scheduler cases pass.
- Existing regressions preserve real committed schedule changes, saved
deadlines, manual/scheduled reservation, retained success and failure
results during unregister, completion/unregister lock order, independent
progress, rollback/outer commit, quiet configured states, drain/resume
and shutdown cleanup.
- The parent's complete seeding/producer run passed 522 cases at its
stated revision. Earlier duplicate reservations, overwritten retained
results, hourly-clock fixture failures and combined-stack CI/setup
failures remain recorded in `doc/idle-plugin-jobs.md`, its dated audit
and the stack ledger; they are not erased or recounted as this
candidate's tests.
- The completion timeout is transaction-local and defaults to thirty
seconds per PostgreSQL statement. It does not bound every
shutdown/network operation or make plugin side effects safe to replay.
No full local repository-suite or deployed host-sleep claim is made.

Commands: server Vitest for the eleven suites above; server
typecheck/build. Fresh exact-head CI, Greptile and Ponytail reviews
remain required before merge.

## Risks

- This uses process-local notifications. Independent SQL writers and
other replicas need their own wake integration.
- A run left unresolved after process loss is not automatically
replayed: the external effect may have occurred. Existing
lifecycle/operator reconciliation must resolve it. Live-process
settlement failures retry without replay.
- Job definitions remain instance-scoped. This PR does not implement
company-scoped job dispatch.
- Cloud's plugin sleep blockers remain. Internal deadlines do not add
durable external wake coverage or prove deployed sleep.
- The shared deadline helper now retains direct lifecycle wakes through
drain; existing coordinator and decision deadline consumers are tested.

## Model Used

OpenAI Codex, GPT-6-based agent, with repository tools, code execution,
and GitHub access. The exact serving model version and context-window
size are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-27 21:07:10 -05:00
DottaandPaperclip 3421366903 fix(server): schedule decision TTL expiry from saved deadlines (#17479) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Open decisions have a saved expiry time.
> - The main maintenance tick scans those times even when none is due.
> - Committed creation and resolution can maintain the actual next
deadline.
> - The existing scheduler can run expiry only when that deadline
arrives.
> - This pull request gives decision TTL expiry its own worker.
> - Terminal history becomes quiet while expiry and continuation
delivery remain recoverable.

## Linked Issues or Issue Description

Targets master after #17478 merged. This is slice 10 of the recurring
server work migration. It owns decision TTL expiry. Target invalidation,
interrupted effects, and retention remain separate migrations. The full
ledger is in `doc/plans/2026-10-24-recurring-server-sleep-stack.md`.

**What existing behavior does this improve?**

Time-based decision expiry in the server.

**Current behavior**

The combined decision maintenance sweep queries expired open decisions
on each tick. It mixes that query with target invalidation, effect
recovery, and continuation delivery.

**Proposed behavior**

Creation, bundle creation, and existing decision updates notify the
shared scheduler. A dedicated worker runs at the first saved open
expiry. Resolving an open decision removes its deadline. The combined
tick opts out of TTL selection while its other duties remain intact.

**Reason and benefit**

A future decision no longer requires repeated TTL queries before its
deadline. Terminal history and an empty decision table leave no
recurring TTL work. No queue table is added.

## What Changed

- Add an expiry owner for bounded due selection, next-deadline lookup,
and expiry settlement. Reuse it for expiry-on-use and the existing
target expiration transition.
- Use the central expiry-only creation notification before insert,
including bundles and caller-owned transactions. Update and terminal
transitions continue to notify expiry and continuation.
- Register the enabled-heartbeat TTL worker before decision
housekeeping. Preserve disabled-heartbeat behavior and the public
combined sweep's default batch limit.
- Honor `PAPERCLIP_DECISIONS_SWEEP_BATCH_SIZE` (default 100). Keep a
fair expiry/ID cursor across failures. Stop between candidates on
shutdown and reconstruct saved work after restart.
- Recheck open status and due TTL during settlement. Merge metadata and
persist activity atomically with expiry. Publish activity and deliver
continuation after commit. Handle exact timestamp equality.
- Continue successful overdue batches after one second through the
existing deadline adapter; preserve thirty-second failure retry and all
default callers.
- Add generated migration `0389_brave_hulk.sql`, an idempotent
concurrent partial index on open expiry and ID. Preserve five generated
snapshots.
- Add 20 real-PostgreSQL lifecycle cases and startup ownership
assertions. Document contracts and limitations in
`doc/idle-decision-expiry.md`.

## Verification

Current candidate: `aa03d2c4c25af4e6c2acf49d3160bbaf6ada030b`. This PR
targets master after #17478 merged. Creation now signals only its real
expiry obligation; terminal updates preserve continuation delivery.

- **117 current cases pass** across all 20 expiry, 54
decision/continuation, 42 startup and one scoped-deadline cases. Server
typecheck/build pass. Separate migrated PostgreSQL fixtures exercise
real transaction and notification behavior.
- Single and bundle creation each failed a regression before the
expiry-only signal repair: open rows unnecessarily notified continuation
recovery. Both now arm their saved expiry without a continuation signal.
Cancelling each row afterward still signals and delivers continuation
normally. The distinction is in the existing central notification owner;
there are no per-route flags or counters.
- **Three new real PostgreSQL regressions failed before repair:** a
configured batch of two was ignored, a successful 101-row backlog waited
for the thirty-second error retry, and an activity insert failure left
the row expired without its audit. All three pass. The audit failure
rolls back expiry; retry records one activity, preserves unrelated
metadata and wakes once.
- The shared deadline adapter has optional `continuationMs`. Only this
owner opts into one-second successful overdue continuation. Other
callers keep the previous default, and errors still retry after thirty
seconds. The archive consumer passes with the option absent.
- Migration `0389_brave_hulk.sql` is regenerated from the parent's
metadata. Parent SQL/journal entries are preserved, five generated
snapshots remain, schema drift and migration numbering/safety pass.
Fresh migration apply and replay pass.
- Lifecycle coverage includes first producer activation, future-deadline
quiet, terminal history, exact equality, commit/rollback, restart,
drain/resume, failed expiry and wake delivery, fair paging through
failed rows at PostgreSQL timestamp precision, shutdown and recovery.
- Failure accounting: the first full repair selection passed 113 cases
and failed a startup assertion that observed a pending follow-up
notification before reconciliation. The assertion now waits for the
exact saved future deadline, then still proves no early expiration.
Earlier fault-fixture and timestamp-cursor failures remain recorded in
[the behavior audit](doc/idle-decision-expiry.md). Extra upstream
embedded-only tests attempted on the parent could not initialize
locally; CI must cover that environment.

Commands: server typecheck/build and Vitest for `decision-expiry-work`,
`decisions-service`, `decision-retention-service`,
`server-startup-feedback-export` and `scoped-deadline-work`; DB
migration checks, snapshot drift and actual migration apply/replay. No
full local repository-suite or deployed sleep/cost claim is made. Fresh
current-head CI and Greptile/Ponytail remain required before merge.

## Risks

- Expiry uses `expiresAt <= now`; the old combined loop skipped exact
equality. The deadline boundary is now covered explicitly.
- TTL expiry records the TTL reason. Target-driven expiry preserves its
own reason. When both causes apply, the first terminal transition wins.
- Supported producers share the application DB signal owner. Independent
SQL writers or multiple active processes need a separate notification
contract. New deadline-edit APIs must use the existing helper.
- The index requires top-level concurrent migration execution. It is
registered with the migration owner and replayed in verification.
- Future in-process deadlines are not external Cloud alarms. Existing
sleep guards remain conservative. This PR does not claim host/database
suspension or measured savings.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact
serving model identifier and context window were not exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-27 18:10:36 -05:00
DottaandPaperclip a63d19816f fix(server): wake decision continuations from committed changes (#17478) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A completed decision can owe a wake to the agent that proposed it.
> - The decision row already stores the pending continuation.
> - The main maintenance tick scans this debt even when no continuation
is ready.
> - Committed changes can activate recovery and leave settled history
quiet.
> - This pull request gives continuation delivery its own worker and
shares foreground delivery ownership.
> - Pending work stays recoverable without an unconditional continuation
scan.

## Linked Issues or Issue Description

Targets master after #17475 merged. This is slice 09 of the recurring
server work migration. Each PR owns one reason for background work. The
remaining inventory is in
`doc/plans/2026-10-24-recurring-server-sleep-stack.md`.

**What existing behavior does this improve?**

Decision continuation delivery and recovery in the server.

**Current behavior**

Foreground paths send a wake and clear a pending flag. The combined
decision sweep scans pending continuations on every maintenance tick. It
can overlap foreground delivery.

**Proposed behavior**

Terminal decision writes notify a dedicated worker through the existing
transaction boundary. Foreground and recovery calls share one in-flight
delivery per decision and DB owner. Recovery re-reads readiness, pages
past failures, and retries only unresolved debt or errors.

**Reason and benefit**

Open decisions, running effects, and delivered history no longer cause
an independent continuation scan. No queue table is added. Decision
expiry, target invalidation, interrupted effects, and retention remain
separate duties.

## What Changed

- Extract continuation readiness, delivery, and paged recovery into the
decision owner. Preserve the wake payload and at-least-once crash
behavior.
- Signal the ten decision update sites before SQL in their existing
transaction scope. Keep effects and wake calls outside those
transactions.
- Explicitly create the continuation obligation on all three
archive-proposal failure settlements, including recovered running rows
without inherited pending metadata.
- Serialize concurrent decision effect execution through final
settlement. Coalesce foreground and worker wakes. Re-read the durable
flag before sending. Fence effect completion to running state so
competing completion cannot restore acknowledged debt.
- Expose an opaque DB work-scope key. Child pools and nested
transactions share that identity; independent roots remain separate.
Existing signal consumers keep their behavior.
- Bound each recovery pass to 100 candidates, retain a fair cursor
across failures, and stop between deliveries on shutdown. Forward the
existing AbortSignal through the shared deadline adapter.
- Register enabled-heartbeat continuation recovery before decision
housekeeping. Remove only its independent scan from the main tick.
Preserve the public combined sweep and disabled-heartbeat behavior.
- Add generated migration `0388_ambitious_micromax.sql`, an idempotent
concurrent partial index for pending continuations, and keep five
generated snapshots.
- Add real producer, transaction, failure, concurrency, paging, quiet,
and shutdown tests. Document ownership and remaining limits in
`doc/idle-decision-continuations.md`.

## Verification

Current candidate: `18f682863cadb91281a86b4da2553a4579ee054a`, normally
integrated with landed archive squash `5785090ff1`. All 31 independent
incoming master files are preserved exactly, including
controller-authority recovery. All 97 decision/startup/deadline cases
plus server types/build pass after the required Runner vendor rebuild.
The preceding full five-suite selection passed 293 cases; the
continuation owner behavior is unchanged by master reconciliation.

- **97 cases pass at this master integration** across decision, startup
and scoped-deadline suites, plus server types/build. The preceding
parent candidate passed the full 293-case
decision/gateway/archive/startup/deadline selection; those results
remain evidence for that revision. All selected cases executed with
separate PostgreSQL fixtures where needed.
- Tenant-driven global scan amplification reproduced six sweeps for six
sequential real cancellations. Existing thirty-second notification
coalescing now permits one recovery sweep per window while retaining
pending work. The first green fixture wrongly expected the intent hold
to clear before coalesced reconciliation; the hold remains and quiet is
checked after settlement. Foreground delivery stays immediate.
- The archive contention test uses a signaled dedicated
single-connection read and two FIFO SQL round trips instead of a fixed
delay. Removing effect coalescing makes it fail with two contender
reads; restoring production code passes. No production test hook was
added.
- **Three new real PostgreSQL regressions failed before repair**:
recovered archive proposals with invalid signatures, mismatched
manifests or archive execution failures and no inherited pending flag.
Each now persists failure, wakes once, acknowledges continuation
delivery and remains quiet on replay. Running-state fencing is
preserved.
- The unmerged continuation index is regenerated as
`0388_ambitious_micromax.sql` after parent migrations 386/387. All
parent journal entries are unchanged, five generated snapshots remain,
numbering/safety and snapshot/schema-drift checks pass. Fresh database
migration apply and replay pass; the concurrent index is valid.
- Existing lifecycle coverage includes foreground/worker coalescing,
child pools, transaction commit/rollback, failed acknowledgement,
restart, drain/resume, shutdown and fair 100-row recovery. Earlier
bounded-pass and concurrent archive-result regressions and their repairs
remain recorded in [the owning behavior
audit](doc/idle-decision-continuations.md).
- Failure accounting: the first combined command used an incorrect
retention fixture variable and skipped 15 cases. The corrected separate
external-PostgreSQL retention run passed all 15. The historical nine
embedded-only work-signal cases were not executed by this selection.
Extra upstream embedded-only suites attempted on the parent could not
initialize locally; they remain CI verification, as recorded in the
parent PR. Earlier reused-fixture, assertion and compile failures remain
in the behavior audit and stack ledger; they are not erased by later
passing checks.

Commands: `pnpm --filter @paperclipai/server typecheck`, `pnpm --filter
@paperclipai/server build`, server Vitest for `decisions-service`,
`decision-retention-service`, `server-startup-feedback-export` and
`scoped-deadline-work`; `pnpm --filter @paperclipai/db
check:migrations`, migration apply/replay, and the generated
snapshot-drift test. Local targeted evidence does not establish a full
local repository run, deployed sleep or fleet savings. Fresh exact-head
CI and both automated reviews remain merge gates.

## Risks

- Notification-driven recovery remains global but is coalesced to thirty
seconds. This bounds request-driven scans; it does not introduce
company-scoped selection. Foreground delivery remains immediate.
- Delivery remains at least once. A crash or acknowledgement failure
after accepting a wake can send it again. Local coalescing is not a
cross-process claim.
- Supported writers share the application DB signal owner. Independent
SQL writers or multiple active server processes need a separate wake
contract. Wake-producing service methods do not gain a new guarantee
when called inside an uncommitted caller transaction.
- The concurrent index needs top-level migration execution; it is
registered with the existing migration owner and can be replayed safely.
- No Cloud sleep blocker or external alarm coverage changes. Worker
quiet does not prove deployed app or database suspension.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact
serving model identifier and context window were not exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-27 17:44:51 -05:00
DottaandPaperclip 5398275d2a fix(server): schedule gateway credential expiry from committed changes (#17450) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The tool gateway stores credentials for sessions and named gateways.
> - Those credentials already contain expiry dates.
> - The server repeatedly scans their tables even when no expiry work
exists.
> - Creation signals and saved deadlines can drive cleanup without idle
scans.
> - This PR registers that work with the existing coordinator.
> - Authentication continues to reject expired credentials independently
of physical cleanup.

## Linked Issues or Issue Description

Refs #17446, #17221, #17274. This is the third sleep-migration slice,
stacked on #17446.

**What problem does this solve?**
Gateway credential cleanup queries PostgreSQL on every maintenance tick,
including empty tables and retained non-expiring tokens.

**What is the proposed solution?**
Signal expiring token/session creation from its transaction. Restore the
earliest saved expiry at startup. Retry only while cleanup or
reconciliation remains outstanding.

**Why is this useful?**
Configured idle gateways no longer create recurring expiry traffic.
Existing credential data remains the durable source of truth.

Related work: #17221 and #17274 change run-token lifetime and cleanup
eligibility. They remain separate. This PR preserves current master
authorization and deletion rules. When those PRs are combined, update
the deadline query and signals to match their new operator-token expiry
predicate; run-token idle markers must not keep this worker scanning.
Retain the separate run-liveness reaper and run both regression suites.
Text mergeability is insufficient evidence for that integration.

## What Changed

- Make the credential test clock follow real scheduled work without
overshooting future-expiry assertions; production behavior is unchanged
by this fixture follow-up.

- Coalesce sequential credential notifications within the former cleanup
cadence, preserve loaded expiry deadlines and immediate authentication
rejection, and bound scheduled deletion to 500 rows per pass.

- Added transaction-aware intent before the two existing credential
inserts. Non-expiring named tokens do not signal expiry work.
- Added earliest-expiry selection in the gateway service and a narrow
registration wrapper for the shared deadline helper.
- Removed gateway credential cleanup from the aggregate maintenance
tick. Startup reconstructs saved deadlines. Heartbeat-disabled instances
retain their startup-only cleanup behavior.
- Preserved expiry/revocation checks, run authorization, audit behavior,
and the existing 500-token cleanup batch limit.
- Added real PostgreSQL tests and a lifecycle/producer document.
Tool-action review recovery remains a separate slice.

- Drain all expired session/token pages in explicit and
scheduler-disabled startup cleanup, while retaining one-page scheduled
passes and authentication enforcement.

## Verification

- At `907098cd3656bac59c60220c73b085d0349d4b5f`, all 290 Runner
transport tests pass locally (including all seven prepared-environment
variants). CI run `40570815360` attempt 1 had one provider-drain
settlement failure in the `close-attachment` case; the identical source
passed the preceding CI lane. A single failed-job rerun is in progress.
The failure remains recorded; no shutdown assertion or production
behavior was changed to suppress it.

- Master landing reconciliation at
`907098cd3656bac59c60220c73b085d0349d4b5f` after #17446 merged: all
production and test files are byte-identical to
`24318d9b99ee7f044a864ff3981fc1929bf3e509`, where the 150-case
acceptance and server types passed. Only commit ancestry and the plan's
merged status changed. Fresh CI/review is pending on this head.

- Parent reconciliation at `24318d9b99ee7f044a864ff3981fc1929bf3e509`:
150 affected tests pass across gateway credential expiry, the complete
tool gateway suite, custom-image worker and startup. The accepted
PostgreSQL runs skip no cases. Server types pass. Existing direct tests
cover sequential-write coalescing and saved-deadline precedence; the
older Ponytail finding needs a fresh review. Remote CI is pending.

Current head: `10c85a5eaa288867aa2997a27aee5f675c2a713d`. This remains a
single-duty PR stacked on #17446.

- Integrated pinned master `8548173fc9efdc0fa07644fb92dcc0b6851a95e5`
with normal merges. Every branch passed generated-schema checks and its
target-branch migration-order gate. Existing SQL is unchanged; the 25
stack migrations follow master’s new migration.
- Duty-specific producer coverage, lifecycle tests, and earlier failure
records:
[2026-10-08-instance-sleep-and-scheduler-inventory.md](https://github.com/paperclipai/paperclip/blob/10c85a5eaa288867aa2997a27aee5f675c2a713d/doc/plans/2026-10-08-instance-sleep-and-scheduler-inventory.md),
[2026-10-24-recurring-server-sleep-stack.md](https://github.com/paperclipai/paperclip/blob/10c85a5eaa288867aa2997a27aee5f675c2a713d/doc/plans/2026-10-24-recurring-server-sleep-stack.md).
- Combined tip `51e6cbe67979b55b340ea9f0f434c5b5c682ccd6`: **355
startup/cancellation/recovery/dispatch/preservation tests**, **75
connection-management tests**, and production server typechecking pass.
These combined checks do not mean every intermediate PR independently
ran the full repository suite.
- Real master-to-stack database upgrade passes: **379 unchanged prior
migration receipts plus 25 new migrations**, with the gateway
responsibility fixture preserved.
- Owning-branch checks: 84 runner installation/settlement cases plus
runner typechecking and generated TypeScript/Rust profile agreement; 253
GitHub review workflow cases. Stop-attribution and held-worker readiness
regressions failed before their fixes.
- Failure accounting: the initial combined run passed 349 cases and
failed six idle assertions in a reused preservation database containing
old cancelled runs. Repeating the combined command with a fresh
preservation database passed. Earlier invalid project selection, merge
import/type errors, and the interrupted full local suite are not counted
as passing checks.
- **Latest-head CI and review remain required.** Prior CI findings
include GitHub idle observation, route teardown, attention cleanup,
touch-picker E2E, workspace health, and voice-cleanup timing; local
passes do not erase those failures. No full local repository-suite,
deployed sleep, fleet cost, or live Copilot qualification claim is made.

## Risks

- Shared-owner signals cover supported local producers. Independent SQL
writers or another replica need a separate ingress contract; startup
recovery is not live discovery.
- This is scheduling only. The open run-token lifetime PRs need semantic
integration as described above before combining their behavior.
- Expiry cleanup can wait during drain; authentication still enforces
expiry on use. Existing Cloud safety gates are preserved. Local
deadlines do not prove external wake coverage or deployed suspension.
- Revocation does not move expiry. Non-expiring tokens remain retained
by design; this PR does not invent a retention deadline.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact
serving model identifier and context window were not exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Latest-head CI is now green after one evidence-backed retry. Greptile is
5/5 and all review threads are resolved.

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-27 14:58:11 -05:00
DottaandPaperclip 623f1dc289 fix(server): schedule secret-proposal expiry from committed changes (#17446) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agents can propose secrets and bindings for approval.
> - Pending proposals already contain their expiry and encrypted value.
> - The server scans for expired proposals on every maintenance tick,
even when none exist.
> - Committed proposal changes and saved expiry times provide the
required triggers.
> - This PR uses those triggers and preserves scrubbing, approval
checks, and recovery.
> - Empty and completed proposal history no longer causes recurring
expiry queries.

## Linked Issues or Issue Description

Refs #17443. This is the second PR in the sleep migration stack and
targets its branch.

**What problem does this solve?**
The periodic proposal scan contacts PostgreSQL when no proposal needs
maintenance. That obstructs idle database behavior across tenant
instances.

**What is the proposed solution?**
Use transaction-aware notifications for creation and resolution.
Reconstruct the earliest pending expiry at startup. Run cleanup at that
deadline, or retry while reconciliation remains unfinished.

**Why is this useful?**
It removes an idle scan while retaining security checks and physical
ciphertext cleanup. No queue table or migration is required.

## What Changed

- Added intent notifications before secret/binding creation, approval,
terminal transitions, and linked interaction resolution. The existing
outer transaction owns settlement.
- Kept expiry selection, ciphertext scrubbing, dependent rejection, and
authorization in the secret-proposal service.
- Registered a startup and deadline worker in place of the aggregate
maintenance sweep. Empty/terminal states disarm; overdue batches and
failures retain retries.
- Reused shared scoped deadline dispatch for secret expiry. The service
owns company-filtered SQL and expiry policy. Custom-image expiry
continues to use `registerDeadlineWork`.
- Added real PostgreSQL producer, outer-transaction, concurrent-write,
expiry, restart, interaction resolution, and drain/resume tests. Updated
startup wiring fixtures and documented the supported writer/lifecycle
contract.

## Verification

- Landing reconciliation at `c10eb1bcf752b058713e05d3345e5cf38dd3fa72`:
160 tests pass across the complete secret-proposal, scoped deadline,
company batch, coordinator, startup, custom-image service and
custom-image worker suites; no cases skipped in the accepted runs.
Server typecheck passes. This integrates master after #17443 landed;
fresh remote CI and Ponytail review are required before merge.

Current head: `5bbe17ad9ebc4573b08ed1a9323644ef9608ca09`. This remains a
single-duty PR stacked on #17443.

- Integrated pinned master `8548173fc9efdc0fa07644fb92dcc0b6851a95e5`
with normal merges. Every branch passed generated-schema checks and its
target-branch migration-order gate. Existing SQL is unchanged; the 25
stack migrations follow master’s new migration.
- Duty-specific producer coverage, lifecycle tests, and earlier failure
records:
[2026-10-08-instance-sleep-and-scheduler-inventory.md](https://github.com/paperclipai/paperclip/blob/5bbe17ad9ebc4573b08ed1a9323644ef9608ca09/doc/plans/2026-10-08-instance-sleep-and-scheduler-inventory.md),
[2026-10-24-recurring-server-sleep-stack.md](https://github.com/paperclipai/paperclip/blob/5bbe17ad9ebc4573b08ed1a9323644ef9608ca09/doc/plans/2026-10-24-recurring-server-sleep-stack.md).
- Combined tip `51e6cbe67979b55b340ea9f0f434c5b5c682ccd6`: **355
startup/cancellation/recovery/dispatch/preservation tests**, **75
connection-management tests**, and production server typechecking pass.
These combined checks do not mean every intermediate PR independently
ran the full repository suite.
- Real master-to-stack database upgrade passes: **379 unchanged prior
migration receipts plus 25 new migrations**, with the gateway
responsibility fixture preserved.
- Owning-branch checks: 84 runner installation/settlement cases plus
runner typechecking and generated TypeScript/Rust profile agreement; 253
GitHub review workflow cases. Stop-attribution and held-worker readiness
regressions failed before their fixes.
- Failure accounting: the initial combined run passed 349 cases and
failed six idle assertions in a reused preservation database containing
old cancelled runs. Repeating the combined command with a fresh
preservation database passed. Earlier invalid project selection, merge
import/type errors, and the interrupted full local suite are not counted
as passing checks.
- **Latest-head CI and review remain required.** Prior CI findings
include GitHub idle observation, route teardown, attention cleanup,
touch-picker E2E, workspace health, and voice-cleanup timing; local
passes do not erase those failures. No full local repository-suite,
deployed sleep, fleet cost, or live Copilot qualification claim is made.

## Risks

- Notifications cover supported writers sharing the database signal
owner. Independent SQL writers and other process roots require a
separate ingress contract; startup recovery alone is not live replica
support.
- The shared coordinator and scoped worker also serve later stack
duties. Their defaults remain intact; existing voice, chat, native, and
startup consumers are included in integrated verification.
- Startup now checks overdue proposals immediately. During drain,
physical cleanup waits; approval still rejects expired proposals
synchronously.
- Each due company pass remains bounded to 100 rows. Failed scopes back
off from 30 seconds to five minutes; healthy overdue work keeps
draining. Self-generated write hints coalesce and do not erase a failed
sweep's backoff.
- Heartbeat-disabled behavior and Cloud sleep-safety gates remain
unchanged. This does not establish deployed suspension or external wake
coverage. Tool review and gateway token cleanup remain separate slices.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact
serving model identifier and context window were not exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-27 14:36:28 -05:00