Jannik/PaperClipAI
1
0 Fork 0
mirror of https://github.com/paperclipai/paperclip.git synced 2026-11-01 06:42:06 +01:00
Code Issues Packages Projects Releases Wiki Activity
6,040 Commits 1,347 Branches 2,200 Tags
8ff812cdde481c215ef4feb767993a834de88e92
Commit Graph
2176 Commits
Author SHA1 Message Date
DottaandPaperclip 8ff812cdde feat(chat): deliver proactive task messages to agent chats (#18374) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agent Chat gives each person a persistent conversation with each
agent.
> - Tasks and routines produce results outside that conversation.
> - Those runs need a safe way to bring a requested result back to the
person.
> - The server must select the agent's own chat and check the source
task's access rules.
> - This pull request adds proactive chat messages and unread
indicators.
> - People can see the result, open its source task, and reply in the
same chat.

## Linked Issues or Issue Description

**Subsystem affected**

Server, native runner tools, shared contracts, database, and UI.

**Problem or motivation**

An agent can report progress on a task or comment on an authorized task.
It cannot send a routine result to its own standing chat with the
responsible person. A morning review routine therefore cannot leave a
visible digest in that person's chat.

**Proposed solution**

Add `send_chat_message({body, idempotencyKey})` and `POST
/api/agents/me/chat/messages`. The server selects the company, agent,
and responsible person from the active run. It saves an attributed
comment and audit record in one transaction. The chat shows a
source-task link and per-person unread state.

**Alternatives considered**

A destination parameter on a generic comment tool would expose
unnecessary routing authority. A new delivery worker would add another
durable delivery path. This change uses a dedicated tool and the
existing comment and live-event paths.

**Roadmap alignment**

This extends the experimental Agent Chat capability in `ROADMAP.md`. The
existing `enableAgentChat` setting controls it. Related public work:
Refs #13282, Refs #13283, and Refs #14408. Searches for proactive chat
and unread chat found no duplicate PR.

## What Changed

- Add a focused messaging service shared by the native tool and HTTP
route. It checks current run ownership, execution mode, recipient
membership, task access, and task privacy.
- Reuse the canonical company/agent/person conversation. Save the
comment and audit record together. Keep task completion and chat
execution separate from delivery.
- Add source-task provenance and source-scoped retry keys. Identical
retries return the saved receipt. Changed content returns a conflict.
- Enforce the conversation's fixed agent at the shared comment-write
boundary. Keep normal reply cancellation and session checks.
- Add chat unread state, accessible agent dots, aggregate Chat
indicators, and visible-message read cutoffs. Refresh through existing
live events.
- Add the request-key constraint, native tool contracts, documentation,
capability maps, stories, and regression coverage.

## Verification

- Local delivery/security proof: all 38 proactive-message cases pass.
Eight credential-scope regressions and four quota regressions fail
against the prior source. The repaired delivery and canonical
cross-issue cap suites pass all 49 cases; all 31 OpenAPI tests, server
typecheck, and capability-map checks pass. The shared per-run cap covers
native and HTTP sends, mixed ordinary mutations, concurrent last-slot
contention, and retry/recovery without double charging.
- Credential tests preserve the authenticated person and authorizer;
reject bridge/test credentials and persisted skill-test runs after
task-mode changes; and recheck API-key revocation, scope, and
responsible person after destination locking. The earlier
delivery/native-authority suites passed all 74 cases before the four
quota additions.
- Four PostgreSQL contention cases cover native and HTTP sends plus
retries during a native chat mutation. All four deadlock against the
earlier source and pass with destination-before-source/run locking.
Recovery and human-reply delivery enqueue only the human comment; a
proactive digest never wakes the chat agent.
- Passed 108 companion conversation, native-authority, and OpenAPI
tests; 112 runner catalog/workflow-report tests; 47 completion-delivery
tests; and 25 project-handoff tests. Focused runner tests cover native
tool availability without generic API tools.
- All 327 affected UI tests, UI typecheck, token gates, and catalog
checks pass. The read hook retries at most three times, respects server
retry delay, coalesces only seen cutoffs, recovers on reconnect, and
cancels on scope changes.
- Real browser journeys with a deterministic process agent verify
routine-to-chat delivery, first-chat creation, unread state across
reload, the source link, secondary chat navigation, and a follow-up
reply referring to the digest. This does not qualify live GitHub
retrieval or model output quality.
- The Storybook showcase uses production components with deterministic
network fixtures. Rendered interactions pass for the landing, ordinary
read success, transient read recovery, and persistent failure after
bounded retries. Both Chat navigation variants update from two unread
chats to one while only Kodi's dot clears. Mobile and light variants
were reviewed. The existing Coordinator Recents label contrast finding
is unchanged; no Linux visual baseline archive exists.
- Current master's independent sandbox migration retains 0418.
Regenerated the feature migration as 0419 from the combined schema and
retained exactly the newest five snapshots; numbering/safety, replay,
and schema-drift checks pass.
- Workspace typecheck and build passed before PR preparation; canonical
server/UI typechecks passed again on the combined tree. Migration
`0419_slippery_impossible_man.sql` passes safety and schema drift checks
on disposable PostgreSQL, including interrupted index repair, concurrent
migrators, one-time owner read-state backfill, preserved receipts, and
two SQL replays. A default five-second local snapshot comparison timed
out; a diagnostic 30-second run passed without drift.
- Earlier failures are retained in the record: full local `pnpm
test:run` and broad shards encountered embedded-PostgreSQL startup,
timing, and provider-recovery failures. Thirteen selected
external-channel cases pass in isolation on the original and changed
sources; an unordered comment-row assertion now explicitly orders both
reads. The earlier route registry, native tool count, runner catalog,
and accessible Chat badge-name failures were fixed and passed full
remote CI on `920b3bb7db`.
- Repeated managed-skill server8 failures came from CI's newer synthetic
merge: its Cloud guard correctly rejected the old local-disk fixture.
Master independently repaired the fixture in #18385; this branch keeps
that repair unchanged. All 57 managed-skill and Cloud prerequisite cases
pass on the combined tree, and that fixture has no remaining PR diff.
- Current reviewed head `dc16f314739a6d977bed21cb1446d0e80a541578`:
Greptile 5/5 with no findings, Superagent security assessment passed,
advisory Paperclip Review 5/5 with no findings, and zero open review
threads. All 55 current-head check runs pass, with five expected skips
and the advisory review neutral at 5/5; Snyk also passes. No checks are
pending or failing. Protected squash auto-merge is enabled and is
waiting for required code-owner approval. A normal exact-head merge was
refused by the base-branch policy; no admin bypass was used. The
previous head's Superagent quota finding is fixed and covered above.

## Risks

- Chat remains company-visible. Delivery rejects a source task with
private-task, project, or ancestor restrictions.
- Read state uses a comment timestamp cutoff. The server retains
database precision and never moves a read receipt backward.
- The new unique constraint changes comment query planning. An existing
regression now uses an explicit row order. The migration builds the
unique index concurrently, then takes a brief table lock to attach it.
It uses the existing top-level migration and interrupted-index recovery
path.
- The upgrade backfills missing chat-owner read-state rows at the latest
existing agent comment. Existing read receipts remain intact. New chats
can still be unread before the human participates.
- Quota observation commits atomically with delivery. A rejected send
rolls back its attempted denial audit row and any first-chat creation;
the canonical warning remains in server logs, and previously spent quota
remains enforced. Identical/conflicting retries and receipt recovery
spend no extra slot.
- This version delivers in-app only. It adds no external-channel, push,
or email delivery.
- Local full-suite acceptance remains limited by the failures recorded
above. The feature stays behind `enableAgentChat`.

## Model Used

OpenAI Codex. PR preparation and CI work used `gpt-6.1-sol` with tool
use and code execution. The earlier implementation also used Codex, but
its exact model ID was not recorded in the visible session. The
context-window size was not provided.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run focused tests locally and they pass; the broad local
failures are recorded above
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-30 21:39:07 -05:00
Devin FoleyandPaperclip 87ae20ae65 fix(server): reconcile failed sandbox cleanup with signed absence (#18407) 
Reconcile failed ephemeral Cloud leases through a signed cleanup-only absence observation when local historical inspection is unavailable. Preserve failure outcomes, exact resource bindings, retention and row-version fences; grant no run-continuation authority.

All 54 current-head checks/statuses passed. Greptile is 5/5, Medium, with no findings or unresolved threads. The 38 focused tests pass; both new reconciliation cases fail without the fallback. Isolated baseline and branch idle suites each pass all 225 cases. The original broader local teardown timeout is retained in the PR, and the full local run remains in progress.

Cloud protocol support is already serving in staging. Exact candidate composition and hosted provider sleep/wake qualification follow this merge. Production has not been changed.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-30 17:04:04 -07:00
Nicky Leach 9e904ee519 perf(server): overlap independent heartbeat preparation reads (#18372) 2026-10-30 16:49:20 -07:00
DottaandPaperclip 91261d5a03 fix(server): preserve budget history for deleted resources (#18389) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Budget views show saved policies, incidents and spend for each
company.
> - A policy can remain after its agent or project has been deleted.
> - The overview currently stops when it cannot resolve that old
resource.
> - A current agent's budget then becomes unreadable during normal
setup.
> - This pull request preserves the historical row in display summaries.
> - Admission and policy changes still require a real company-owned
resource.

## Linked Issues or Issue Description

**What happened?**

The budget overview throws `Agent not found` when a saved policy
references a deleted agent. One old resource prevents the company from
reading its current budgets. The same problem affects historical project
policies and incident summaries.

**Expected behavior**

The overview should retain historical policy and incident IDs and spend.
It should display `Deleted agent` or `Deleted project` for a missing
resource. Read access must not grant permission to execute work or
change a budget for that resource.

**Steps to reproduce**

1. Create an agent and a budget policy for it.
2. Permanently delete the agent after its other dependencies have been
cleared.
3. Read the company's budget overview. The existing service throws
instead of returning its remaining current and historical rows.

**Paperclip version or commit**

The isolated repair starts from merged commit
`d9354bbfc35ef74333e90e596ce21dc8e59149b9` and integrates the exact
merged baseline test fix `8595efc20bda3ce20a9fbbf34a5d156a6fcf6d57`. It
also integrates the actual merged Antigravity auth fix
`18b1637b6371a6a6adade857e2a73654ee626fef` (#18254). The diff against
the selected merged parent remains two budget files.

**Related work**

Earlier reports and implementations: #11058 and #4333. This extracts the
product prerequisite from #18351 so it can qualify independently of that
journey's other dependencies. The change contains only the budget
service repair and four database regression cases.

## What Changed

- Resolve deleted agent and project scopes only when building display
summaries. Preserve their original IDs, policies, incidents and
accounting records.
- Reject a saved policy or incident that resolves to another company's
resource. Propagate other read failures.
- Keep missing-company resolution, work admission and policy changes
strict. Validate company, policy and incident scope bindings before
either incident action. A budget raise requires a live company-owned
resource. Historical dismissal may proceed only for a genuinely deleted
owned agent/project, and must reject live foreign or mismatched scopes.
- Add four canonical PostgreSQL regression cases: deleted agent history,
deleted project history, missing-scope admission and budget raises with
safe historical dismissal, and foreign/mismatched references. The
history cases also verify current ledger spend and unchanged stored
rows.

## Verification

- Real-service RED/GREEN against official PGlite with every canonical
migration: the original overview throws for deleted history; the repair
returns the historical IDs, incidents and spend while retaining current
budgets.
- Mutation regression RED/GREEN: missing-resource budget raises must
reject before writes; owned deleted-scope dismissal must reject its
linked pending approval, retain policy/agent/ledger data and incident
history, and remove the incident from the active list. Live foreign and
mismatched scopes reject both actions without mutations.
- Current integrated head `b3434f5ad05fd3067379352700c36bfe826179c4`:
all four canonical PostgreSQL regressions pass in [server
2/12](https://github.com/paperclipai/paperclip/actions/runs/41142588583/job/124361194077),
and all five existing budget-service tests pass in [server
7/12](https://github.com/paperclipai/paperclip/actions/runs/41142588583/job/124361194080).
Both use the CI merge against selected parent `18b1637`.
- Clean-build declaration typing passes in the current CI typecheck
gate. The earlier local source-alias no-emit check also passed,
including the canonical actor argument. The local worktree borrows
compiled dependencies from a sibling checkout; a later real-service
import was blocked by its stale `sandboxCreationOperations` export, so
clean-build CI is authoritative for the final integration. No production
workaround was made.
- `git diff --check` passes. Broad repository typecheck, tests and build
run in current-head CI rather than repeated locally. Native local
PostgreSQL could not start because this host exhausted its semaphore
pool; no global IPC settings or unrelated processes were changed.
- One unchanged Runner lane retry was authorized after a probable
intermittent close-settlement fixture failure. The [original
failure](https://github.com/paperclipai/paperclip/actions/runs/41142588583/job/124360234111)
occurred in `omitted-context` cleanup. A [fresh passing Linux
lane](https://github.com/paperclipai/paperclip/actions/runs/41142552195/job/124360110386)
passed the same case and all 1,998 tests with byte-identical entire
Runner and adapter-utils trees, lockfile, root package, scripts and
workflows, against the same merged base. The fixture has a five-second
close allowance with a 2.5-second suspension reserve; a host stall is
not proven. No budget path or changed close input was found. The [single
targeted
retry](https://github.com/paperclipai/paperclip/actions/runs/41142588583/job/124365812609)
passed the same case in 1,034 ms and all 1,998 lane tests; dependent
verification also passed. All 57 current PR checks are terminal and
green.
- The product helper requires all current CI and clean review before
merge. Actual QA budget-overview proof will run on the combined final
app before hiring qualification. No deployed success is claimed. The
older `b4863dd` artifact was never deployed; selected parent `18b1637`
adds sandbox/lifecycle changes and migration 0418, so whole-app identity
is not claimed. The budget module and relevant budget contracts remain
unchanged, and final hiring requires the combined final app with
explicit source identities.

## Risks

Budget summaries now return historical rows whose resources have been
deleted. These rows use explicit display names and preserve their
accounting data. They do not create a resource or relax work admission,
authorization or budget grants. Historical incident dismissal changes
only that incident and its linked approval; it does not resume work or
raise a budget. Cross-company references still fail closed. There is no
schema migration and no new background work.

## Model Used

OpenAI Codex, GPT-6 family. The session does not expose a more specific
backend model ID or fixed context window. Used reasoning, repository
tools, TypeScript and database test execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes (the
PR description documents the display-only history and strict-mutation
contract)
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-30 18:36:36 -05:00
DottaandPaperclip 45a3a92e6d fix(environments): persist sandbox ownership before creation (#18357) 
## Thinking Path

> - Paperclip manages companies and the agents that do their work.
> - Remote execution creates provider resources that need a durable
owner.
> - The server could create a sandbox before it saved its lease.
> - A failed lease write could leave cleanup data only on the old web
server disk.
> - This change saves a company-scoped operation before creation and
transfers ownership atomically.
> - Recovery finds the original Daytona resource or records an explicit
manual reconciliation outcome.

## Linked Issues or Issue Description

**What happened?**
A sandbox could exist without a lease after a database failure or
company rebind. Failed cleanup recording could then create a local
fallback file. Replacing that web server could lose the cleanup
reference.

**Expected behavior**
A failed admission write makes no provider call. An interrupted creation
retains a committed operation. Recovery must not create a second sandbox
or destroy an attached lease.

**Steps to reproduce**
1. Run the disposable PostgreSQL cases in
`server/src/__tests__/sandbox-creation-operations.test.ts`.
2. Reject the lease insert after the controlled provider creates its
resource.
3. Start a new receiver process with a different empty web disk. It must
find the same resource and create one cleanup owner.

Related ownership report: Refs #14672. This change addresses new
allocations before lease persistence. It does not replace existing lease
recovery.

## What Changed

- Add an idempotent migration and a small creation-operation table.
Retain original company, environment and run identities across deletion.
- Commit creation intent before run, ad-hoc lease, and plugin task
replacement calls. Fence duplicate requests and ambiguous lease commits.
- Retain a known returned task resource ID in the existing operation
before attachment. Preserve manual reconciliation and never delete an
ambiguously attached resource.
- Keep company and run authority checks and atomic lease transfer with
the lease owner.
- Reuse Daytona names, labels, read-only inspection and claimed cleanup.
Keep provider payload handling in the provider.
- Retain nonsecret credential identity hashes. Refuse recovery lookup
under a changed credential value.
- Record manual reconciliation for missing/deleted or explicitly
inactive saved credentials, versions, and provider vaults. Keep
temporary resolution failures retryable. Never reactivate a disabled
credential.
- Use transaction signals and concrete deadlines. Failed records back
off independently. Manual reconciliation stays durable and quiet.
- Remove new orphan-file fallback writes from the covered paths. Keep
legacy file replay and its tests for cutover.
- Replace obsolete fallback-producer assertions with normal-entrypoint
durability tests. Keep cleanup, diagnostics and warm handoff coverage.

## Verification

- [Independent current-head
verification](https://github.com/paperclipai/paperclip/pull/18357#issuecomment-6534744320)
passed 190/190 combined tests, 40/40 incoming TestOps tests, full
workspace typecheck, and build. PR head
`958af986130f25e136065ad30d1fdd843c46255c` has tree
`e6c3e5b02182d5941a70ccf5bf55638df3c2e28d`, identical to tested combined
commit `b9d7a9c2e9b30f0153685fb24a5f70b56132201f`. Real PostgreSQL,
compiled Rust Runner and controlled object providers were used;
external-S3 integration remains outside this local proof.
- Current head `958af986130f25e136065ad30d1fdd843c46255c` normally
merges master `8595efc20bda3ce20a9fbbf34a5d156a6fcf6d57`. The sole
conflict was the maintenance test: a parameterized case now preserves
both legacy-local `local_storage` and valid shared-S3
`provider_mismatch` checks, including the parent local receipt, exact
version identity, missing-snapshot/no-seed refusal, and unchanged
persisted row. The three master TestOps files remain intact. No
production, schema, migration, or package changes were made after the
fully qualified prior head `0b4d05505e`. Independent review approved the
resolution; the full managed-skill-content file and Cloud guard controls
pass 58/58 (105.11 seconds). [Required PR CI passed on this exact merge
head, first
attempt](https://github.com/paperclipai/paperclip/actions/runs/41136987412):
54 successful checks and three intended skips. The additional Paperclip
review is neutral at 4/5; two external Superagent checks (Contributor
trust and Security Scan) remain in progress with zero annotations and no
assessment, so they are not passing scan evidence. Greptile reviewed
exact head `958af986130f25e136065ad30d1fdd843c46255c` at 5/5; the final
thread audit has zero unresolved threads. All completed evidence below
retains its recorded revision scope. The additional Paperclip Review on
`958af98613` is neutral at 4/5 with a Ship verdict and no must-fix or
should-fix item. Its optional observation is that a generic create
rejection remains pending/manual even when an operator believes
allocation did not occur. This change preserves uncertainty because the
generic create-error contract does not prove provider absence; no new
error classification or provider-specific compensation was added.

- [Independent final-source
verification](https://github.com/paperclipai/paperclip/pull/18357#issuecomment-6534328247)
passed all 189 real-PostgreSQL/Runner and controlled-fixture cases, full
workspace typecheck, and build. The integration commit
`c8411a8528386f4824b340ec5fdaaed1efb9ea69` has the same source tree as
final PR head `0b4d05505e` (tree
`bda65efc14b4c904d0f589154db77bb160641c8c`). Controlled object-provider
coverage passed; the separately guarded external-S3 suite was not run
because the bounded local MinIO pull was unavailable. This includes the
two final compatibility fixes.
- The first final-head CI attempt exposed one E2E timing failure in
`disconnect-write-recovery.spec.ts:181`: the title stayed unchanged
through the 15-second session-recovery poll. The same original case
passed in 13.8 seconds on immediately prior head `7ec06f77b7`, and all
UI/session/E2E source is identical across those heads. The failure log
and CI artifacts are retained; no cause is claimed. One failed-job-only
retry is authorized on this evidence, without changing a timeout,
assertion, or production behavior. [The single failed-job retry
completed
successfully](https://github.com/paperclipai/paperclip/actions/runs/41133619710/attempts/2):
all 35 E2E shard cases passed (6.6 minutes), including the original
session-recovery case in 14.0 seconds. The initial failure remains
disclosed and its cause remains unproven.
- Final head `0b4d05505ee0aa72f2981cdff74761406960ec73` adds two
independently reviewed integration fixes after the complete 7ec CI
matrix exposed two deterministic failures. Server 9/12 failed legacy
local-storage maintenance because the CLI reached the new Cloud server
startup guard; server 5/12 found the Daytona package version did not
match its bumped manifest. The failed run is retained at
https://github.com/paperclipai/paperclip/actions/runs/41131204203. It is
not passing evidence and was not blindly retried.
- Maintenance now constructs the existing explicit storage dependency
after loading the required persisted Cloud snapshot, and passes it to
both maintenance owners. It can verify legacy local storage before
provider cutover while normal Cloud server startup remains guarded. The
original case reproduced red locally, then the complete
managed-skill-content and managed-cloud-prerequisites files passed 57/57
(32.05 seconds), including missing-snapshot refusal, unchanged stored
configuration, and existing server guard controls. Daytona package
version now matches manifest 0.1.10; the complete bundled package
identity file passed 14/14 (3.98 seconds), and explicit
manifest/lifecycle controls passed 2/2 (328 intentionally filtered; 3.42
seconds). An initial workspace-filter command matched no project and is
not counted as pass evidence. The final diff changes no sandbox runtime
or schema behavior. [Full CI is green on this exact final
head](https://github.com/paperclipai/paperclip/actions/runs/41133619710/attempts/2):
56 successful checks, three intended skips, one additional
incomplete/neutral Paperclip Review, and no pending or failed checks.
The neutral review is not a passing assessment. [Greptile reviewed the
exact final head at
5/5](https://github.com/paperclipai/paperclip/pull/18357#issuecomment-6529396969),
with no new actionable issues and all review threads resolved.
- Final combined head `7ec06f77b7c14a745c569d18401f16df3d26b626`
includes master managed-skills migration 0417 and shared-storage
prerequisites from `198218bf55` through normal merges. Sandbox creation
moves to generated `0418_curved_lady_mastermind.sql`; its idempotent SQL
is byte-for-byte unchanged. The regenerated 0418 snapshot contains both
sandbox creation operations and the master skill mutation receipts, and
points to the master 0417 snapshot. Master SQL and journal history
remain intact. Exactly five snapshots remain, 0414 through 0418. The
only sandbox-specific changes since the approved credential policy are
migration metadata/number and the matching test/document references; the
sandbox runtime files are unchanged.
- [Independent final combined
verification](https://github.com/paperclipai/paperclip/pull/18357#issuecomment-6533960026)
passed on integration revision `e6f05b4cf0cd`, containing final sandbox
head `7ec06f77b7` and actual master `198218bf55`: migration/declaration
checks, all 158 real-PostgreSQL/Runner and controlled-fixture cases (251
seconds), full workspace typecheck (81 seconds), and build (35 seconds).
- Final combined migration proof: fresh real PostgreSQL has both tables.
A separate original-0417 sandbox table with a retained pending row
survives two applications of the renamed 0418 SQL unchanged, with the
expected primary/index and foreign-key structure. Migration
numbering/safety checks pass. The normal runtime migration reapply case
passes 1/1 (20 intentionally filtered cases; 46.69 seconds). The first
temporary qualification harness stopped before any database assertion
because its external `.ts` file selected CommonJS; the `.mts` harness
corrected that setup and passed. Independent review approved the exact
metadata and reference diff. Earlier full suite and live-provider
results below retain their stated revision scope.

- Credential-policy head `c606d473b03709f27577a55a1dfc0b0d3b4359fd`
handles the remaining inactive-credential review warning. The existing
boundary accepts only concrete secret-owner 422 codes and exact
no-details disabled/coming-soon vault messages. Unknown 422,
authorization errors, provider wrappers, and temporary database/provider
failures still propagate. Independent source review approved this
three-file correction. No generic resolver, schema, provider protocol,
retry cap, or activation behavior changed.
- Creation/cleanup qualification at `c606d473b0`: all 39 cases pass
against real PostgreSQL (81.27 seconds). The new cases use actual
disabled/archived saved secrets, a valid disabled version, and provider
vaults created ready before normal API updates disable them or mark them
coming soon. Each remains manual through restart/repeated scans, with
zero provider lookup and no retry deadline. The independent plugin idle
blocker remains intact. The temporary 503 control retains retry
ownership and later recovers one cleanup owner. Before this correction,
all five inactive-state cases remained pending; that failed evidence is
retained. The prior 18-case known-ID task proof below belongs to
`237144fcf6`, whose helper and task path are unchanged by this
correction.

- Known-ID head `237144fcf69a249c0e0310563428c0f55ed6d55c` also
addresses the independent task-replacement reliability warning. The
existing creation operation now saves only the known nonsecret returned
resource ID before expiry validation and attachment. The write is scoped
to operation/company/original environment and allows only a missing or
matching ID. It changes no ownership state and copies no resolved
configuration, credentials, or provider payload. Independent source
review approved this four-file fix.
- The complete real-PostgreSQL task suite passes 18/18 (9.52 seconds).
Rejected attachments retain the ID through manual reconciliation and
repeated scans. A real committed attachment followed by an injected lost
response retains the same active lease and attached operation, with no
release or second create. Same-ID retries work; conflicting IDs and
foreign companies cannot change it. The initial three regression cases
failed because the ID was absent, before the fix.
- The shipped Daytona manifest does not declare task support, so it
cannot enter this generic replacement path. The dispatcher requires both
declared task support and an environmentTask worker method. Generic
drivers without creation lookup still use manual reconciliation, with a
known returned ID retained when the database write can commit. No
automatic lookup capability or unconditional compensation was added.

- Previous boundary-fix head `8ba77def151ea85eeae2db7693254db7f87673b0`
fixes the fresh review finding about missing credentials. Only known
secret-owner 404 results at the recovery secret-resolution boundary
select the existing manual outcome. Provider inspection and transient
database/provider errors remain retryable. Independent source review
approved this change.
- Secret-boundary focused acceptance at `8ba77def15`: all 34 cases pass
against isolated real PostgreSQL. The complete creation suite passes
16/16 (100.75 seconds); the complete cleanup coordinator suite passes
18/18 (99.83 seconds). The new cases use normal saved-secret creation
and deletion, confirm zero provider lookup, restart/repeated scans, no
automatic deadline, preserved independent plugin idle blocking, and no
additional idle blocker after the controlled worker supplies its quiet
proof. A temporary 503 resolution failure preserves pending ownership
and later recovers one cleanup owner. No additional live provider
resource was created.
- Failed attempts remain part of the evidence. The initial red run
reproduced the pending-operation bug and also exposed a numeric-deadline
test harness mistake, which was corrected. A later combined run reported
an existing replacement-receiver failure and a new idle-fixture
assertion failure; it was interrupted while the next test file remained
in imports. An isolated diagnostic confirmed that the manual state was
correct but the installed fixture plugin correctly blocked sleep without
its independent quiet proof. The corrected test retains that blocker
assertion before supplying the controlled worker witness. The unchanged
replacement-receiver case passes in the final complete 16-case run; its
earlier failure is not counted as a pass or assigned a proven cause. No
timeout or production idle guard was relaxed.
- Greptile reviewed previous combined head
`7ec06f77b7c14a745c569d18401f16df3d26b626` at 5/5, with no unresolved
review threads. Its CI produced the two concrete integration failures
addressed above. The earlier CI and live evidence below remain
historical evidence, not a claim that the current full gates have
completed.

- Master integration update: merged `dbff3747b8` and then latest master
`76fde891cc` without rewriting branch history. Only `doc/DATABASE.md`
conflicted; the sandbox ownership, object-backed run evidence, and cloud
configuration sections all remain. The master merges changed no sandbox
implementation, migration, or provider code. Migration numbering/safety
checks pass and exactly five schema snapshots remain. At merge head
`f5d23d4a59828835d01c150f83fa2645fb2d3bec`, the two focused
creation/coordinator suites pass all 32 cases against isolated real
PostgreSQL (64.57 seconds). Those merge-head checks precede the
missing-credential fix below. The earlier live Daytona proof still
covers the unchanged normal creation/lookup/teardown path; the new
missing-credential branch is qualified with real PostgreSQL and
controlled provider hooks.

The optional Superagent Security Scan was inconclusive on prior head
`7ec06f77b7`: its pinned PR merge-base checkout failed or timed out,
with a neutral conclusion and zero annotations. An earlier scan could
not start within 120 seconds; a bounded rerun request returned HTTP 404.
These scanner failures are not passing security-scan evidence. On final
head `0b4d05505e`, the additional Paperclip Review check is neutral: its
assigned review execution did not complete and produced no validated
assessment. Fresh Greptile 5/5 and independent source review are
separate evidence. Previous-head required CI, independent review, and
the live Daytona qualification below passed.

- [Complete previous-head
CI](https://github.com/paperclipai/paperclip/actions/runs/41105726052)
passed at `0b5cbb763ab8faa25aa0ff1d8a405cdc0d3db409`: 53 successful
checks and three intended skips, including typecheck, full tests, build
and E2E. Greptile is 5/5 on this head and both original findings are
resolved.

- Initial head: 200 PostgreSQL runtime, creation and coordinator tests
passed. The corrected head also passes all 32 creation and coordinator
tests, including deletion before/during lookup, a failed lookup after
deletion, restart, settled polling and actual idle-safety checks. Cases
cover failed admission, failed lease writes, concurrent recovery, lost
provider replies, ambiguous transfer retries, company rebind,
environment deletion, stopped runs, lookup backoff, restart, drain and
settled SQL quiet.
- Separate receiver processes: passed. A controlled HTTP provider
retains the resource while three independent receiver processes use two
empty web disks. Recovery creates one cleanup owner and never repeats
creation.
- Fresh PostgreSQL migration and two additional applications: passed.
The pending creation row survives unchanged.
- Server TypeScript and SDK/shared build checks pass after the
corrections. Migration numbering/safety checks passed. Exactly five
schema snapshots remain.
- Daytona adapter: all 330 tests passed, including stable-name lookup,
ownership checks, deletion and the updated manifest version. An earlier
run exposed the version expectation and it was corrected. Previous-head
CI passed.
- Live Daytona acceptance passed on `0b5cbb763a`: isolated real
PostgreSQL, normal runtime acquisition through independent receiver
processes, actual Daytona plugin/SDK 0.203.0, one disposable Ubuntu
24.04 sandbox (1 CPU / 1 GiB RAM / 1 GiB disk). A forced lease insert
failure retained the committed operation; a fresh receiver with empty
disk found the same sandbox, repeated recovery retained one cleanup
owner, and the actual claimed cleanup sweep expired the lease and
resolved the operation. Independent SDK lookup returned typed not-found;
ordinary recorded-ID inspection returned destroyed. This successful
attempt made one create, one recovery lookup, and one teardown. Two
earlier failed harness/provider-confirmation attempts are disclosed in
the [live acceptance
record](https://github.com/paperclipai/paperclip/pull/18357#issuecomment-6530313054).
The coordinator independently queried all three exact resource IDs
through the SDK and confirmed typed not-found for each.
- Complete CI-affected heartbeat/plugin warm-task suites passed 21/21
after test corrections. An unchanged workspace exposure case passed in
isolation after the prior CI shard reported an unhandled readiness
rejection. Complete previous-head CI and refreshed review passed.

## Risks

- Deleted companies retain an explicit manual reconciliation outcome.
Recovery does not retry missing credentials or insert a lease for a
deleted company.
- Only Daytona declares automatic creation lookup. Other uncertain
providers retain `needs_reconciliation`. A missing creation name alone
cannot prove that a late allocation will never appear.
- Pending operations use the existing process-local signal and startup
recovery model. This does not add cross-replica wake delivery or prove
deployed sleep.
- Legacy direct provider probes and custom-image setup use separate
lifecycles. They are outside this replacement of the run-lease
orphan-file fallback. The normal Test path that acquires a runtime lease
is covered.
- Existing orphan files still require the separate cutover inventory and
settlement. This change does not delete them.
- The schema change is additive. The new SQL can be reapplied.
Historical SQL and journal entries are preserved; only the oldest
generated snapshot is pruned.

## Model Used

OpenAI `gpt-6.1-sol`, with tool use, code execution and repository
review. The harness did not report the context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge


The required branch status checks (`ci / verify`, `ci / e2e`) and all
test/build jobs passed on this head. The non-required Superagent
Contributor trust and Security Scan checks are still running without
findings; they are not claimed as passes. No required check or branch
rule is overridden.

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-30 17:38:23 -05:00
Devin FoleyandPaperclip 81ecaf0654 fix(server): bound failed lifecycle verification (#18383) 
Failed lifecycle verification could allocate another test environment every minute indefinitely. Count failed host verification tests separately from claims, persist a repair wait for confirmed configuration failures or three generic failed tests, and clear it through authorized retry or execution configuration changes.

Preserve typed temporary execution-target failures through adapter checks so timeouts, transport failures and interrupted probes keep automatic recovery. Retain existing readiness, authorization, termination and cleanup gates. Document the policy and add real PostgreSQL persistence and actual OpenCode adapter regressions.

Verified 398 focused tests (one Windows-only test skipped on macOS), repository-wide typecheck and build, structural checks, fully green current-head CI and Greptile 5/5 with no unresolved comments.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-30 15:29:43 -07:00
DottaandPaperclip 8595efc20b fix(evals): support official TestOps job metadata (#18385) 
## Thinking Path

> - Paperclip manages AI agents and verifies real work through Product
E2E tests.
> - The fixed Chat feature workflow uses TestOps to select and report
one backlog case.
> - Official registration exposes a workflow-query URL and a Branch
parameter.
> - The existing reader accepts only workflow-file or numeric URLs, and
the admin launcher requires an operation parameter.
> - These assumptions reject valid official metadata before the feature
can run.
> - This pull request supports the exact observed form while preserving
source, selection, and once-only execution checks.

## Linked Issues or Issue Description

Refs: #18322, #18375.

**What happened?**
The second zero-model bootstrap stopped with a generic job-mismatch
code. Ordinary TestOps UI inspection then showed an official
workflow-query URL outside the accepted literals and a Branch-only job.
The exact first failed predicate in that bootstrap remains unknown.

**Expected behavior**
Accept that exact fixed-workflow URL. Preserve the existing master
Branch value when the job has no operation parameter. Keep rejecting
foreign URLs, unknown parameters, wrong branches, and conflicting
operation IDs before mutation.

**Steps to reproduce**
Pass the official query URL and Branch-only metadata through the
existing bootstrap, selection, and admin-launch owners. The new pure
controls exercise those actual helpers without a CLI, backend, or model
call.

**Paperclip version or commit**
This change starts from `0e15c92073a09e50df99190a37f69a86286b989b` after
#18375.

**Deployment mode**
GitHub Actions and the protected TestOps reporting environment.

## What Changed

- Share one exact URL predicate across bootstrap, evaluation selection,
and admin preparation. Add only the observed official workflow-query
literal.
- Support Branch-only, operation-only, and Branch-plus-operation jobs
with strict definition IDs, names, values, and duplicate checks.
- Preserve the exact master Branch value. Set an existing operation
parameter to `evaluate`; otherwise use the fixed workflow's existing
default.
- Use the same checked parameter vector for job and plan writes and
their readbacks.
- Report distinct fixed bootstrap predicate codes without response
values.
- Document when the protected operation-parameter ID is required.
- Align one existing Cloud-maintenance test with the inherited
managed-Cloud S3 policy. Preserve the same-version local-storage proof
in the parent, use valid synthetic Cloud settings in the child, and
check the expected provider mismatch without S3 requests. Retain the
no-seed and unchanged persisted-row assertions.

## Verification

- URL controls: three failures on the parent; all pass with the
correction.
- Branch controls: two failures and one correct rejection on the
predecessor; all pass with the correction.
- `node --test tests/runner-e2e/chat-feature-testops-api.test.mjs`: 40
owning pure/mock controls pass on Node 26.4.0.
- Two MJS syntax checks and `git diff --check` pass. TypeScript noEmit
was not run for this MJS/documentation change.
- The first [normal CI
run](https://github.com/paperclipai/paperclip/actions/runs/41131094974)
failed only in the managed-skill Cloud fixture and its dependent
verification gate: 46 jobs passed, two failed, and one skipped. The
tested merge included a new upstream policy that rejects the test's
local-disk Cloud configuration. The exact caught runtime exception
remains unknown.
- Three pure controls using the actual policy validator pass: the old
invalid Cloud fixture rejects, the new synthetic S3 fixture admits, and
an invalid master key rejects. An initial tsx wrapper IPC permission
error executed zero controls and remains a separate setup failure.
- The PostgreSQL/maintenance-child test was not run locally. On new-head
AWS CI, `managed-skill-content.test.ts` passed all 17 tests in server
shard 9.
- [Normal
CI](https://github.com/paperclipai/paperclip/actions/runs/41133876636)
is green on exact head `b76d5dd40d7711f1b5f6b6316b1114c0aae02d57`: 48
jobs passed and one skipped. Both required `ci / e2e` and `ci / verify`
pass.
- Greptile reviewed this exact head at 5/5 with no actionable findings
or unresolved review threads.
- No actual CLI, backend, model, provider, app, database, build, or
bootstrap was run to validate this correction locally.

## Risks

The exact original bootstrap predicate is still unknown. This fixes a
demonstrated metadata false negative and preserves the prior failed run.
Actual backend parameter types and integration dispatch still need
verification. Unknown metadata stops before the once marker and writes.
This PR changes no workflow, credential, case, model, grader, clock, or
private-evidence policy. It does not claim a passing feature result.

## Model Used

OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) with reasoning, code editing, tool
use, and pure test review. The runtime does not expose the exact
context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-30 17:04:59 -05:00
DottaandPaperclip 198218bf55 fix(config): require shared storage for managed cloud (#18350) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Managed Cloud instances must retain shared objects and secrets
across server replacement.
> - Implicit storage defaults can select local disk or an unintended
bucket.
> - Cloud runtime settings now come from a canonical database snapshot.
> - This pull request validates bootstrap prerequisites before database
setup and effective S3 settings after snapshot resolution.
> - The benefit is an actionable refusal before local signing state or
HTTP setup, while valid canonical Cloud settings work.

## Linked Issues or Issue Description

**What happened?**

A managed Cloud instance could start with local disk storage, an
implicit bucket, or a local key file. After canonical Cloud
configuration landed, checking only environment or file values would
also reject a valid database bucket. Shared schema defaults could make
an omitted raw bucket appear explicit.

**Expected behavior**

Managed Cloud requires external PostgreSQL, a stable environment master
key, and effective S3 settings with a nonblank bucket. The canonical
snapshot remains the configuration authority. Invalid configuration
refuses startup before signing or HTTP effects. Self-hosted defaults
remain compatible.

**Steps to reproduce**

1. Select managed Cloud with its tenant token or managed configuration
document.
2. Leave storage as local disk, or omit the S3 bucket from a raw
canonical snapshot.
3. Start the server. Startup must refuse. An explicit valid bucket in
the canonical snapshot must work without a file or environment bucket.

**Paperclip version or commit**

Current base: `4edd10804e2b42941aa2038e90c362c13a0b98dc`.

**Deployment mode**

Managed Cloud. Ordinary self-hosted defaults remain unchanged.

Related PRs: #18335 owns canonical Cloud runtime configuration. #18327
owns S3 run logs. #18352 owns credential persistence. #18347 owns
cutover checks. This PR adds configuration prerequisites to those
existing boundaries.

## What Changed

- Validate external PostgreSQL and a stable environment master key at
the Cloud bootstrap boundary, before database setup.
- Validate effective S3 settings after canonical snapshot publication
and before Cloud signing, runtime consumers, or HTTP setup.
- Require a nonblank bucket in the Cloud snapshot schema. Reject omitted
or blank buckets in every legacy Cloud file before one-time import
normalization, even when an environment bucket is supplied. Keep shared
self-hosted schema defaults.
- Preserve typed bucket provenance at the existing file-read boundary.
Accept explicit canonical bucket names, including `paperclip`, and
environment settings when the snapshot is null.
- Validate effective legacy storage with the existing resolver before
seeding. A rejected candidate leaves no canonical rows, so a corrected
source file can be imported on retry. Environment overlays are checked
but never persisted.
- Add real PostgreSQL snapshot and migration-retry tests, startup
refusal tests, and environment/file compatibility controls. Document
that Cloud always requires its stable environment key, even with an
external default secrets provider.

## Verification

- Latest focused acceptance passed 199/199 in six files (54.29 seconds),
using the command below plus
`server/src/__tests__/cloud-configuration-migration-postgres.test.ts`.
Real PostgreSQL proves rejected storage leaves all three migration
authority rows absent, then a corrected legacy file imports on retry. An
external-default-secrets control proves the deliberate unconditional
Cloud master-key contract.
- Before the peer correction, the frozen product diff passed 178 tests
in five files: managed-cloud-prerequisites, cloud-runtime-config,
server-startup-feedback-export, config-file, and
cloud-configuration-readers. Command: `pnpm exec vitest run
server/src/__tests__/{managed-cloud-prerequisites,cloud-runtime-config,server-startup-feedback-export,config-file,cloud-configuration-readers}.test.ts
--maxWorkers=1 --no-file-parallelism`. Real PostgreSQL used a qualified
POSIX test-only PG18.6 override because the host cannot supply the stock
runtime's SysV semaphore allocation.
- Coverage includes raw database and legacy omitted/blank bucket
rejection, valid database and environment buckets, null snapshots,
explicit default-named buckets, self-hosted compatibility, and refusal
before local signing mutation.
- The real AWS SDK configured-provider control uses a simulated loopback
S3 service. It writes, reads from a fresh provider, and deletes an
object. It also reads an encrypted credential with the stable
environment key. This does not qualify production cloud permissions.
- `git diff --check` passed. Final merges add unrelated master changes
from #18241, #18363, and #18370; the focused storage product diff is
unchanged.
- Peer review reproduced a fresh legacy local-disk import that gained
the default bucket before an S3 environment overlay. The importer now
requires a raw explicit bucket for every legacy Cloud file. Six controls
cover omission/blank values with and without an environment bucket; the
full focused rerun passed 184/184 across all five files in 194.89
seconds.
- Initial revised startup testing passed 167/168: one fixture omitted
the database URL and failed before its intended key assertion. The
corrected final run passed 178/178. An initial direct server typecheck
failed because Runner declarations were not built; this is a setup
failure, not a compile pass. Integration verification rebuilds
dependencies before full checks.
- Prior head `fae1e4aa` passed complete CI with 53 successes, two
intended skips, and Greptile 5/5.
`de71345f379cecb71dbfb219f35e14f942c10a3f` then passed complete CI (55
successes, three workflow skips, and a neutral external review), with
Greptile 5/5 and zero unresolved threads. The external review identified
the preseed rejection problem; its correction is committed at
`75e04d0d77e19dcef0a8cb070b6b518f0f173b14`. The final merge head is
`19b809ea2823a6a4e5ceb48ef2c2bac4721733d6`; [Fresh complete
CI](https://github.com/paperclipai/paperclip/actions/runs/41127309762)
passed on that head: 55 successful checks, three workflow skips, and one
neutral external review. Greptile reviewed this exact head at 5/5 with
no actionable issues. Both earlier external review threads are resolved
with the tested fixes. Earlier infrastructure retries remain documented
in [the first
diagnosis](https://github.com/paperclipai/paperclip/pull/18350#issuecomment-6529964690)
and [the runner shutdown
evidence](https://github.com/paperclipai/paperclip/pull/18350#issuecomment-6530364769).
Earlier missing SDK build and fixture metadata failures were repaired
before those passes.
- Current frozen-head CI first attempt failed one unchanged Antigravity
authentication success case in Runner Vitest 1/2 (`outcome=failure`,
1994 tests passed). The same file passed 11/11 locally in 2.33 seconds,
and identical Runner source passed the prior complete CI run. Attempt 2
retried only failed jobs (the Runner shard and dependent aggregate).
Both passed without a source change. No second retry was used. The exact
timing cause is not established. [Failed
job](https://github.com/paperclipai/paperclip/actions/runs/41127309762/job/124309181778).
- No production data, buckets, or live credentials were used. Complete
local repository suites are not duplicated in this worktree;
current-head required CI supplies the full suite gate.

## Risks

- Managed deployments with local disk, omitted buckets, file-only keys,
or missing external database URLs now refuse startup. Operators must
supply valid prerequisites before rollout.
- Required deployment environment overlays must remain stable across
replacements. For example, a saved local-disk provider with an explicit
bucket can use an environment S3 provider; removing that required
overlay correctly refuses startup until the environment or canonical
configuration is corrected. Environment settings are never silently
copied into the canonical row.
- Effective legacy storage must also pass before first seeding. A bad
candidate does not create authority rows. Existing saved rows remain
authoritative and can require environment overrides or a canonical
configuration change if they are invalid.
- A present legacy Cloud file needs an explicit raw bucket even if its
saved provider is local disk or its environment supplies a bucket.
Correct the file before one-time import, or use the existing
absent-file/null-snapshot environment-only setup. Environment overlays
are not persisted as operator choices.
- A previously stored canonical nonblank `paperclip` bucket is
authoritative. Prior normalization erased whether that value was
originally explicit. This change cannot recover that provenance and does
not reject existing nonblank canonical values.
- Validation proves configuration shape, not cloud permissions, database
reachability, or key compatibility with existing encrypted data. Keep
the master key stable across replacements.
- Cloud snapshot JSON parsing is stricter; omitted or blank raw S3
bucket values need correction. No database migration or provider
operation changes are added. These prerequisites do not prove that every
server persistence path is disposable.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

OpenAI Codex, exact model ID `gpt-6.1-sol`. Context window: unknown. The
model used reasoning, repository tools, code execution, and test
execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-30 16:21:46 -05:00
DottaandPaperclip 0e15c92073 feat(skills): make managed content independent of web-server disk (#18334) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agents use installed skills and exact version pins during execution.
> - Managed skills still depended on mutable directories on the web
server.
> - A replacement server could lose acknowledged edits or remove an
unassigned skill whose directory was absent.
> - This pull request makes immutable versions the authority for managed
content and installed repository reads.
> - The server can rebuild delivery files from durable content after
replacement.
> - This establishes skills readiness for ephemeral web servers. General
server draining and workspace recovery remain separate work.

## Linked Issues or Issue Description

**Subsystem affected**

Skills services, shared contracts, database receipts, Skill Studio, and
runtime delivery.

**Problem or motivation**

A fresh web server must discover, read, edit, and deliver protected
installed skills using the same database, object storage, secrets, and
supported application artifacts. It must not need its predecessor's
files or process memory.

**Proposed solution**

Use `managed` ownership with a null source locator. Read installed
content by immutable version. Prepare packages before a short
publication transaction. Persist execution selections before
materialization. Recover exact pinned runtime assets and reject deletion
while active or recovering pins exist.

**Alternatives considered**

Reconstructing a mutable installed directory preserves the old
source-of-truth ambiguity. Permanent dual writes create two competing
authorities. This change uses one managed model after a bounded legacy
conversion.

**Roadmap alignment**

This improves the existing Skills Manager, Studio, version history, and
Cloud execution capabilities. The maintainer supplied the implementation
contract. This does not add a separate skills product or deploy fleet
changes.

Related work: #18082 supplies immutable package storage. #17273
addresses edits outside the API for linked local skills; this PR keeps
that source category separate. #2648 is a closed earlier
ephemeral-filesystem proposal.

## What Changed

- Add managed ownership and focused content publication and execution
materialization modules. Keep the archive format and package storage
tables.
- Publish managed create, save, delete, fork, rename, restore, and
catalog content with version checks. Add durable create receipts.
Preserve file-save receipt responses.
- Return file version identities. Keep Studio drafts after a conflict
and provide an explicit reload action. A delayed reload cannot overwrite
newer typing or navigation.
- Persist exact runtime selections, including empty selections. Rebuild
pinned assets and verify their admitted digests on a replacement server.
- Preserve source access on forks and disconnected copies. Keep the
existing workspace policy category for managed skills.
- Add bounded conversion, eligibility, and independent verification
phases. Load persisted Cloud configuration in the standalone CLI without
seeding or changing it. Reject partial current versions against
installed inventories; recover only complete saved content or the exact
source. Document storage migration blockers and the compatible rollback
floor.
- Add separate-process PostgreSQL/S3 qualification and update domain,
retention, policy, and UI regressions.
- Pass the plugin worker manager into Studio execution, admit restricted
`skill_test` mode through native Runner contracts, and preserve admitted
scope across native and REST tools. Native tests cannot create ordinary
tasks, browse company data, or use external tools.
- Select the exact durable Studio revision even when unassigned or Beta
skills is off. Persist Studio output and audit atomically with native
task completion, preserving the result on replay.

## Verification

- Final revision `13e8af480b88ba6fea5737bac4bd81104d604792` passed full
CI: all 49 workflow jobs completed successfully ([run
41127396098](https://github.com/paperclipai/paperclip/actions/runs/41127396098)).
This includes typecheck, test suites, Runner checks, browser checks and
build. Greptile review 15 is 5/5 on that same revision, with zero
unresolved threads. The PR is ready for review.
- Local workspace typecheck and build passed after integrating master.
Focused checks passed: 33 execution/completion cases, 52 Runner HTTP
cases, 16 managed-content cases, 17 Studio cases, the new real-process
CLI regression, map checks and UI token gates. The CLI regression proves
inspection does not seed missing Cloud configuration. Server typecheck
passed after that fix.
- The independent-process PostgreSQL/S3 matrix passed through normal
HTTP skill routes. It covers A → B → C replacement, lost commit
responses, upload crashes, concurrent saves, binary contents and
executable delivery. Metadata listing and warm selected delivery retain
zero object downloads. Local timing measurements are recorded; they do
not establish hosted latency.
- Hosted A → B → C passed on the owned disposable test tenant with
external Postgres, S3 and a real remote Paperclip Computer. Predecessor
temporary homes were absent. C read and executed B's acknowledged
managed save. New runs used repository v1 while v2 was installed. Hashes
matched an independently checked exact upstream commit.
- Compatible image rollback to `6518b323` passed. Studio retained the
unassigned fork's C save. A fresh remote run executed the newer managed
content and historical repository pin. The actual shell event confirmed
expected hashes, stdout, exit 0 and mode 0555.
- Runtime candidate `9200e3fcb4` deployed successfully ([Cloud run
41124147876](https://github.com/paperclipai/paperclip-cloud/actions/runs/41124147876)).
The final Studio test delivered only the unassigned fork's exact v3,
executed its script and persisted a successful output snapshot.
Independent command-event inspection confirmed all three hashes, stdout,
exit 0 and mode 0555. All 14 installations were version-backed.
Independent verification checked all 25 versions: 23 remote objects, two
bundled database snapshots, zero blockers.
- The deployed image is `9200e3fcb4`. Later master integration, the
client-only delayed-reload draft guard and the CLI bootstrap have
separate automated proof. The CLI bootstrap was also exercised against
the tenant's stored configuration and storage through the same read-only
services. This PR does not claim that the deployed image is the final PR
revision.
- Live Studio checks exposed worker wiring, admission, access scope,
exact selection and completion-publication gaps. They are fixed. Failed
attempts remain in the evidence. The full local suite exceeded 80
minutes under host contention and was stopped. Later combined/isolated
local attempts also timed out; they are not counted as passes. Full CI
covers the integrated suites. Earlier CI found an obsolete restart-scope
fixture, which was corrected and passed all 52 HTTP cases.
- Reproduction and migration:
[`doc/skill-ephemeral-web-servers.md`](doc/skill-ephemeral-web-servers.md).
Source revisions, deployments, versions, run IDs, hashes and failure
history:
[`doc/plans/2026-10-30-ephemeral-web-skills.md`](doc/plans/2026-10-30-ephemeral-web-skills.md).

## Risks

- Legacy conversion requires stopped admission and retired
filesystem-authoritative writers. Pre-conversion binaries are not
supported rollback targets afterward.
- Cloud readiness requires remote object storage. Local provider
receipts and provider mismatches remain blockers until objects are
copied and verified.
- Linked local sources retain filesystem semantics and are outside the
cloud guarantee.
- Cold reads still fetch the existing archive format. Warm runtime
preparation retains the verified-cache path.
- Bundled reconciliation preserves an installed head. Existing release
snapshots remain selectable. An older server must not replace a newer
head with its shipped files.
- No production volumes, fleet defaults, or general deployment
coordination change in this PR.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, repository tools, code
execution, and browser use. The exact backend model ID and
context-window size are not exposed in this session. No subagents were
used.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-30 16:18:27 -05:00
Nicky LeachandPaperclip fd008979f5 test: wait for the evidence expiry worker to settle (#18241) 
## Thinking Path

> - Paperclip coordinates AI work and retains execution evidence.
> - Evidence retention uses an asynchronous worker and a shared
scheduler.
> - The retention test checked the next deadline as soon as an archive
disappeared.
> - File removal can finish before the worker publishes that deadline.
> - This pull request waits for worker settlement before asserting its
next wake.
> - The test keeps its deletion, busy-session preservation, and
quiet-worker checks.

## Linked Issues or Issue Description

**What happened?**

CI for #18239 failed in `native-harness-backup-expiration.test.ts`. The
healthy-expiry test observed the file removal while `nextWakeAt()` was
still null. Subtracting the current time then produced a large negative
number.

**Expected behavior**

The retention test should check the deadline after the asynchronous
worker has settled.

**Steps to reproduce**

Run the server test shard with a slow filesystem completion. The
deletion predicate can become true before scheduler publication
finishes.

## What Changed

- Keep the existing file-removal wait open until tracked work is idle
and the expected busy-session retry deadline is registered.
- Keep the held archive and quiet-worker assertions.

## Verification

- Original CI failure: [server shard
3/12](https://github.com/paperclipai/paperclip/actions/runs/40976031169/job/123804262266).
- The targeted test passes with its real PostgreSQL fixture: `node
node_modules/vitest/vitest.mjs run --project @paperclipai/server
server/src/__tests__/native-harness-backup-expiration.test.ts -t 'lets
healthy saved expiry preempt' --maxWorkers 1`.
- After rebasing onto master, the focused retention test passes again
with its real PostgreSQL fixture. The PR now changes one test, with
seven insertions and two deletions.
- Master independently fixed the authentication timeout and
GitHub-recovery checkpoint race. Their redundant commits were dropped
during rebase.
- [Full GitHub
CI](https://github.com/paperclipai/paperclip/actions/runs/40984842413)
passes on `bbaa7c3c384f6c220ce2c979f0e9b5d01ddb1f77`: all 48 jobs
succeeded. Standard Greptile is 5/5 on that head, and GitHub reports the
PR clean.
- Broader coverage runs in GitHub CI. No local full build or full test
suite was run for this assertion-only change.

## Risks

Low risk. The retention test waits for the condition it already asserts.
The existing deletion, busy-session preservation, future-deadline, and
quiet-worker checks remain in place. No production code or time budget
changes.

## Model Used

OpenAI GPT-6 through Codex, with repository inspection, terminal tools,
and code execution. The serving variant and context window size are not
exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-30 13:28:00 -07:00
DottaandPaperclip 76fde891cc Add bounded recovery file inventory before server cutover (#18347) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Accounting, credential recovery, and sandbox cleanup can retain
legacy files on an instance volume.
> - Replacing that volume while files remain can lose unresolved work.
> - There is no verified production backlog, so an empty installation
needs no migration.
> - The recovery owners must decide whether each record is valid and
settled.
> - This pull request adds a bounded file inventory and an optional
cutover gate.
> - The benefit is a safe check that preserves obligations without
reading secret material.

## Linked Issues or Issue Description

**Problem or motivation**

Operators need to check the actual affected instance directories before
replacing a web server. A skipped corrupt file or failed read must not
appear clear.

**Proposed solution**

Add a Node script that inspects only the three legacy recovery
directories for an explicitly chosen instance. Default mode reports
counts. The optional check mode blocks when files or uncertainty remain.

**Alternatives considered**

A bulk migration system has no established backlog to migrate.
Reimplementing the owner parsers could expose credentials and still
would not prove settlement. This check never reads file bytes.

**Roadmap alignment**

This is a small operational prerequisite for safe cloud server
replacement. It does not add a recovery backend or change resource
ownership.

Related recovery-owner changes: #18354 (accounting), #18352
(credentials), and #18357 (sandbox ownership). This inventory remains
independently usable with existing files.

## What Changed

- Add a pure classification policy and a bounded filesystem boundary.
- Add `node scripts/recovery-cutover.mjs --instance-root
/absolute/instance` with optional `--check`, directory override, and
entry budget.
- Report observed lower-bound counts and safe status summaries. Do not
print paths, filenames, identifiers, raw errors, or file contents.
- Add one dependency-free CLI acceptance suite and a Vitest bridge so
the same cases run in normal server CI.
- Document existing owner entry points, override selection, stopped
producers, and retained-volume requirements.

## Verification

- `node --test scripts/recovery-cutover.test.mjs`: 9 cases pass. The
suite starts the actual CLI with synthetic legacy directories.
- Pending receipts, nested credential files, orphan records, malformed
bytes, temporary files, unreadable files, unsafe paths, symlinks, read
failures, bounds, and configured overrides all block safely.
- Empty or absent recovery directories clear. Source file bytes remain
unchanged. Report-only mode and check-mode exit codes are tested.
- `pnpm exec vitest run server/src/__tests__/recovery-cutover.test.ts
--pool=forks --maxWorkers=1`: the normal CI bridge passes and runs the
same 9 cases.
- Cached frozen dependency installation, script syntax checks, and `git
diff --check` pass. The lockfile is unchanged.
- Independent source review found the read-only boundary sound.
- [Complete current-head
CI](https://github.com/paperclipai/paperclip/actions/runs/41097614666)
passed at `bf5cc7f067041c10cd0294ff26756678da943976`: 53 successful
checks and two intended skips, including typecheck, full tests, build,
and E2E. Greptile is 5/5 with no review threads.
- One failed-job retry was needed for an existing heartbeat timing
failure in server shard 4. The relevant heartbeat source and tests match
the base branch. Local diagnostic runs had 61 pass/2 fail and 7 pass/1
fail, with changing timing failures; neither run is claimed as green.
The bounded retry and all failed evidence are [recorded
here](https://github.com/paperclipai/paperclip/pull/18347#issuecomment-6529629804).
No heartbeat code or test was changed.

## Risks

- The scan is a point-in-time inventory. Stop producers for the final
gate. It is not an atomic drain certificate or protection against
hostile path mutation.
- All files remain unverified. The check cannot certify validity or
Postgres settlement. Existing owners retain that responsibility.
- The scan does not inspect other instances, volumes, Runner delivery,
or database obligations. It does not establish that production cutover
is ready.
- No migration, deletion, decryption, or replay command is added.
Symlink paths are rejected, including system aliases; use canonical
absolute paths.

## Model Used

- OpenAI `gpt-6.1-sol`, with reasoning, tool use, and code execution.
Context window size was not supplied.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-30 15:21:13 -05:00
DottaandPaperclip e54354d9d3 fix(ai-connections): save quota credentials without recovery files (#18352) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Managed AI connections keep the selected account in the secret
store.
> - Server quota refresh can receive a replacement token before its
database save commits.
> - That path currently writes an encrypted recovery file on the web
server.
> - Cloud web replacement should not need that file to use a saved
account.
> - This pull request saves the replacement directly and retries the
same value in memory.
> - The benefit is a smaller cloud storage contract with an explicit
sign-in recovery limit.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Server-owned OpenAI quota refresh and credential persistence during web
replacement.

**Current behavior**

Quota refresh checks local encrypted recovery storage before token
exchange. It writes the replacement to that storage before the secret
transaction commits. A replacement web receiver can therefore need the
old web volume.

**Proposed behavior**

Required database reads and locks must succeed before exchange. The same
logical refresh saves its returned replacement directly, with at most
three save attempts. It checks the current secret after an uncertain
commit response. A process that loses the unsaved replacement can
require sign-in again.

**Reason and benefit**

Postgres remains authoritative. This removes new quota recovery files
without adding another credential vault. Existing files remain readable
for cutover. Local-only Google retirement keeps its existing behavior.
Remote Google subscription sessions remain unsupported.

Related public work: Refs #14056 for managed Codex credential handover.
Searched open and closed credential-refresh pull requests; this change
does not duplicate their provider login work.

## What Changed

- Keep the exchanged replacement and its source generation in the
current refresh operation. Save retries do not repeat OAuth exchange.
- Reconcile a lost commit response from the saved credential. Grant
changes, secret versions, session epochs, selectors, and revocation
prevent a stale save.
- Stop creating or preparing quota recovery files. Preserve existing
record settlement and local Google compatibility.
- Document the v1 sign-in limit and existing Runner
persistence-before-completion contract.
- Add PostgreSQL and controlled-provider regressions for save rollback,
lost replies, revoked or changed authority, receiver process loss, and
pending Runner credential handover.

## Verification

- [Complete current-head
CI](https://github.com/paperclipai/paperclip/actions/runs/41099132607)
passed at `e4ed6551a707c814ddb2395a7358106b1e04857a`: 53 successful
checks and two intended skips. This includes full typecheck, tests,
build, and E2E gates.

- `pnpm -r typecheck` and `pnpm build` passed at
`1ebe75984be9bf19d518728d4ff440a2d7f3dbce`. Later branch changes only
repair tests. Rebase adds an unrelated master UI performance fix; the
credential and Runner handover source did not change.
- Focused credential tests passed: 2 files, 160 tests (145.62s).
Command: `pnpm exec vitest run
server/src/__tests__/ai-connections.test.ts
server/src/__tests__/plugin-task-credential-writeback.test.ts
--maxWorkers=1 --no-file-parallelism`.
- `pnpm test:run` was deliberately interrupted after 17m41s in its first
general-server chat integration phase, before any suite result. The
controller exited 130 and its wrapper exited 1. This is an incomplete
local run, not a full local pass or an observed test failure. The
coordinator stopped duplicate long local work; the complete current-head
CI suite subsequently passed. All owned test and PostgreSQL processes
were confirmed stopped.
- Database tests use real PostgreSQL 18.6 with POSIX semaphores through
a test-only loader. Provider requests use controlled fixtures; no
production OAuth mutation was made.
- Greptile reviewed current head
`e4ed6551a707c814ddb2395a7358106b1e04857a` at 5/5 with zero comments.
- The captured focused-test logs contain none of the replacement-token
fixture values.
- The receiver-loss test kills a separate Node process after one HTTP
token exchange while PostgreSQL blocks its secret insert. A new receiver
then uses the existing sign-in flow.
- The Runner case rejects a real deferred secret commit. Provider
completion remains uncalled until a new receiver commits the retained
credential.
- Initial setup failed because this host exhausted System V semaphore
resources. Another setup attempt lacked built SDK entries. Both were
setup failures, not passes. After setup repair, the first real run
passed 159/160 tests; the remaining test asserted the retired
directory-fsync contract. The replacement test forbids recovery writes
and requires one committed exchange without a file.

## Risks

- If all three saves fail and the producing server process exits, the
unsaved replacement is lost. The next failed use can require sign-in
again. This is the accepted v1 limit.
- Existing unresolved recovery files still require inventory and
settlement before their old volume is removed. This PR does not perform
deployment or delete those records.
- Local Google subscription retirement still needs its existing local
recovery ownership. It is outside the supported remote cloud scope.

## Model Used

OpenAI `gpt-6.1-sol`, with reasoning, repository tools, code execution,
and controlled integration tests. The context window size was not
supplied.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-30 15:20:57 -05:00
DottaandPaperclip dbff3747b8 feat(server): persist run logs and provider traces in S3 (#18327) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Operators use run logs and restricted provider traces to inspect
agent work.
> - Local log and trace files depend on the web server's persistent
volume.
> - That dependency prevents overlapping volume-backed Railway
deployments.
> - This PR reuses the instance S3 store for batched log and trace
payloads with conditional publication.
> - Existing Postgres rows retain their contracts; native export
recovery reuses events already committed before Runner acknowledgement.
> - S3-configured servers automatically copy and verify retained history
before a later deployment cutover.

## Linked Issues or Issue Description

**Subsystem affected**

Server run logs, restricted provider traces, storage providers, and
migration tooling.

**Problem or motivation**

Replacing the web server and its disk must preserve committed log output
and retained traces. The current S3 mirror can repeatedly upload a
growing file. Moving those payloads into Postgres would add database
writes, WAL, replication, and backup work.

**Proposed solution**

Reuse the existing instance storage configuration, S3-compatible
provider and credential chain. Store new run logs under run-logs/ and
opted-in raw traces under provider-traces/, below the existing prefix.
Cloud keeps its existing bucket and bucket-scoped credentials per stack.
Add no new configuration switches. Store immutable payload segments and
bounded binary range indexes directly in S3. Publish each batch with an
ETag-conditional head. Keep existing database ownership and lifecycle
records. Recover canonical native exports from their existing validated
control event source, checkpointing export progress in S3. Add no new
payload writes to Postgres. Preserve local defaults and migrate settled
history with verified byte counts and digests.

**Alternatives considered**

Postgres payload chunks add control-plane load. Repeated whole-file
mirrors amplify transfer work. The first index prototype left superseded
index nodes; the final binary index keeps all normally published nodes
reachable.

**Roadmap alignment**

This is a storage prerequisite for replacing the main web/API server. It
does not enable deployment overlap, detach a volume, or change native
Runner protocol acknowledgements. The maintainer approved this
implementation scope. ROADMAP.md has no duplicate implementation item.

Related PR: #16770 improves the existing in-flight mirror. This PR
preserves that legacy backend and uses the existing instance S3 profile
for new journals.

## What Changed

- Automatically select S3 segments for new logs and opted-in traces when
instance file storage uses S3. Reuse bucket, endpoint, region, prefix,
path style and credentials. Remove separate backend switches. Keep
local-disk instances local.
- Share the existing storage configuration resolver with product files.
Preserve config-file settings, environment precedence and the existing
root prefix. Keep pre-existing run-log mirror settings for legacy
history.
- Add bounded buffering, conditional ownership fencing, immutable range
reads, stable byte offsets, and completion hashing.
- Dispatch all existing log consumers through the recorded backend.
Coalesce output progress writes and publish live log messages after
storage receipts.
- Recover interrupted canonical native exports from their existing
durable source. Exclude separate credential, cleanup, and reprojection
records. Replay missing callback ranges and deduplicate late delivery.
Repair settled archives after controller lease expiry. Bound shared
repair work and stop it when all readers cancel. Serve the published
prefix when best-effort repair fails.
- Add encrypted optional trace channels, replacement reads, durable
deletion tombstones, and deadline-driven local/external expiry cleanup
independently of heartbeat scheduling. Stream historical source and
remote verification. Propagate worker cancellation through trace open,
conditional-write probing, upload, restore and reads. Fence deletion
even before the first object head exists; abort and drain owned
captures.
- Retry transient storage errors, distinguish 409 from failed
preconditions, and backpressure bounded log admission. Track interrupted
trace imports explicitly so ordinary local deletion stays independent of
S3.
- Automatically migrate retained history through the existing delivery
coordinator. Process one log and one trace per batch. Retry unresolved
history with backoff and stop scans after completion. Keep the
inventory/import command as optional diagnosis and repair. Verify source
and remote bytes before switching references. Verify finalized native
size and digest before setting its export cursor. Require ordered source
coverage when completeness metadata is missing. Preserve original trace
expiry. Block active, missing, corrupt, and unproven sources.
- Preserve identity and run-secret redaction in live and recovered
native logs. Register runtime log redaction secrets once through the
existing encrypted registry. Keep agent private keys in their existing
owner.
- Add storage, migration, retention, UTF-8, and route-privacy coverage,
plus operating documentation and qualification tooling.
- Add no schema, payload table, per-record SQL, production dependency,
or lockfile change.

## Verification

- Final fixes at `b3a44c898e` passed all 116 selected startup,
retention, attribution and cancellation controls. The disabled-scheduler
cleanup and concurrent-repair defects reproduced red before the fixes. A
timed-out repair still prevents later storage reads.
- The merged source passed 124 storage and real-Postgres Cloud
configuration tests. These include automatic history migration, existing
environment/file vault keys, source-directory removal, five real-SDK
stalled-request cancellation cases, shared capability-probe cancellation
and multi-chunk trace ordering. Saved Cloud S3 settings work after the
legacy config file is gone.
- Workspace typecheck and full build passed at the merged parent
`19f702d780`. Server typecheck and build passed again at `b3a44c898e`.
- Actual app acceptance: finish a process-agent run through the normal
API, stage its retained local-file input, then restart. Automatic
migration reports one migrated item and zero blockers. Remove the
original web home and restart at `b3a44c898e` with an empty home, the
existing S3 profile and the existing vault key file. The normal log API
preserves all five producer lines and exactly 788 bytes, with SHA-256
`74d9ae469da28bfe89b9b50a0e132aec24fe9e6d686496ed5e4ffa6167151036`. No
log-specific storage settings are present.
- Actual expiry acceptance: with heartbeat scheduling disabled, a trace
remains unexpired after startup and later reaches its fixture deadline.
The worker marks the row deleted and tombstones both S3 channel heads.
The retained run log stays byte-identical. The process fixture emits no
provider protocol frames, so nonempty encrypted payloads are qualified
separately through the real SDK round trips.
- Isolation tests use identical run identities in two stack buckets.
Cleanup in one existing prefix leaves another prefix, another company
and product files intact. Read-only Cloud source inspection confirms a
bucket and bucket-scoped credentials per stack. Four existing Cloud
storage environment-contract tests passed. No Cloud provisioning changed
and no new live vendor permission probe was performed.
- Earlier qualification passed 1,014 affected storage/native tests and
27 privacy-route tests with isolated Docker Postgres. An abrupt process
test commits 1,000 records, receives receipts, dies by SIGKILL and loses
its directory; another process reads every byte and continues. The real
Rust Runner experiment acknowledges 32 records while the export cursor
is zero, then continues the same Runner PID after controller loss. All
64 records export once with exactly 64 existing control rows. Repeated
snapshot/Stop commands completed in 162/81 ms.
- Emulator workload: 1,000 burst records used two batch PUTs. Eight
steady streams produced 240 records with 104 conditional PUTs and 72
GETs over 6.9 seconds. These are synthetic operation counts, not fleet
sizing or provider prices.
- Failure evidence is retained. Compact native events over 64 KiB,
missing file-key support, disabled-scheduler retention and repair
failure handling all reproduced before their fixes. Fixture repairs
addressed cold module setup, localhost/IPC permissions, the S3
bucket-name limit and a mode-0700 crash-test directory. Full local `pnpm
test:run` was stopped after host PostgreSQL semaphore exhaustion;
affected suites used isolated Docker databases instead. Native package
library symlinks were restored with the package's standard hydration
helper.
- The `19f702d780` browser session-recovery case timed out while saving
a task title. All eight browser shards passed on `b3a44c898e`, including
that case, with no UI behavior or assertion change. Fresh-head CI passed
at `b3a44c898e`: 55 successful checks, three skipped checks and one
neutral check. The branch is conflict-free.
- Map and module-boundary checks passed. Current-head Greptile is 5/5
with zero unresolved review threads.

Reproduce the opt-in storage suites with
`PAPERCLIP_LOG_TEST_S3_ENDPOINT` and a dedicated
`PAPERCLIP_RUN_EVIDENCE_MIGRATION_DATABASE_URL` ending in `_test`. See
`doc/run-evidence-storage.md` for operating limits, automatic migration,
optional diagnostics and rollout qualification.

## Risks

- Keep the existing per-stack bucket, prefix and vault key stable across
replacements. Relocating that storage is a separate whole-instance
operation.
- This release automatically changes new log storage for existing
S3-configured instances. Local-disk instances remain local. Canonical
native source replay is qualified with a deterministic real Runner;
actual remote execution and production-provider qualification remain
rollout gates. Generic server-only diagnostics and optional raw trace
tails without a surviving source can be incomplete after a crash. This
does not establish full zero-downtime deployment.
- Before deploying this release, qualify the actual S3 provider. It must
enforce conditional writes and pass read, version, deletion, workload,
and latency qualification. A capability probe rejects providers which
ignore preconditions. Moto is emulator evidence only.
- Dirty output is bounded and unconfirmed until S3 publication. Storage
failure leaves persistence incomplete while task results and usage can
settle.
- Trace expiry is not yet represented by Cloud's external wake contract.
Outstanding trace deadlines conservatively block sleep.
- Automatic history copying requires the original source volume. Keep it
until all retained resources pass verified import and clean-filesystem
reads. Stop old binaries from creating new local-file runs. Missing or
corrupt history remains blocked and visible in server logs. Rollback
requires a compatible reader against the same external state. Ordinary
object deletion does not erase retained historical object versions or
backups.

## Model Used

OpenAI Codex, `gpt-6.1-sol`, with reasoning, tool use, code execution,
and browser verification. The runtime did not expose a context-window
size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-30 15:17:35 -05:00
DottaandPaperclip 6dcf4160dd perf(chat): load recent messages and page history on scroll (#18358) 
## Thinking Path

> - Paperclip lets people manage work done by agents.
> - A standing agent chat can hold months of conversation.
> - The chat route forced full-history mode, bypassing compact task
loading.
> - A cold visit fetched lifetime run summaries and hydrated historical
transcripts.
> - This PR starts with 20 recent messages and loads earlier pages on
upward scroll.
> - It preserves the reader's position, current questions, and draft
while history grows.

## Linked Issues or Issue Description

**What happened?**

Standing agent chats forced full-history loading. A long conversation
fetched full activity and run history before it became readable. Earlier
comments also used an explicit load control.

**Expected behavior**

Open a small recent conversation quickly. Load older messages when the
reader scrolls upward. Show an inline loading state. Keep the visible
message and composer mounted through paging, live updates, and Back
navigation.

**Steps to reproduce**

1. Open a standing agent chat with hundreds of messages and completed
runs.
2. Use a fresh browser query cache.
3. Observe full activity and run reads plus transcript requests for
historical runs.
4. Compare the recent window and upward paging after this change.

Related work: #18312 and #18323 restored compact task and collection
loading. #14848 added reopening of visited agent chats. A GitHub search
found no open PR for this chat-history optimization.

## What Changed

- Return 20 contiguous recent comments for standing chats, with a stable
oldest cursor. Keep the existing compact task exchange contract. Use the
existing company/issue/time index for cursor reads.
- Keep chats compact even if their task view previously expanded
history. Retain observed messages through live catch-up and cache-backed
return visits.
- Keep fetched history pages, acknowledged sends, and the observed
recent cursor separate. Reconcile live replies without losing history
after a send or deleted cursor.
- Hydrate metadata only for newly revealed message/run IDs; preserve
prior batches while paging. Slow-send recovery refreshes the recent
window. Receipts failures have an independent retry.
- Hydrate run summaries only for loaded messages and the latest turn.
Validate batches of at most 100 run IDs and keep the existing
issue/company access rules.
- Reuse hydrated terminal native streams when only history membership
changes. Preserve retry, resume, mode changes, reconnect recovery, and
interrupted recovery reads.
- Load older messages, decision receipts, and task handoffs through
upward scrolling. Support wheel, touch, and keyboard input. Keep pending
questions visible.
- Reserve a status slot while loading. Preserve a logical message anchor
and the mounted composer. Pause failed page reads until an inline Retry.
- Add database, route, controller, and scroll regressions, plus an
opt-in production-build browser rig for long chats on desktop and
mobile. Update design and feature/access documentation.

## Verification

- The real PostgreSQL cold-open suite passes all 12 tests, including
contiguous pages with equal timestamps, complete history, company
denial, and scoped run IDs.
- Activity route and OpenAPI suites pass 53 tests. The route run allowed
local cold-import overhead; CI retains the repository's normal timeouts.
- Three new paging regressions fail against the unchanged source. The
candidate's focused paging/window suite passes 26 tests. Broader UI
verification is recorded below when complete.
- Desktop and mobile browser acceptance covers recent-only startup,
per-frame reveal geometry, delayed and failed older pages, stable
reading anchors and drafts, Back, older comment links, and one persisted
live reply.
- Full workspace typecheck and build passed locally. The final UI source
also passes its typecheck and production build.
- Affected UI suites pass: 212 controller/history/paging tests, 352
thread/scroller tests, 26 native-stream tests, plus metadata batching
and live-cache hydration coverage. The issue mutation/access route suite
passes 179 tests.
- All 22 final desktop/mobile browser journeys pass, including delayed
image growth. The isolated second-page request proof also passes six
repeated desktop/mobile runs. A separate 6,000-message, 3,000-run
fixture passes cold-open and delayed-paging checks on both screen sizes,
still opening 20 messages and loading one 50-message page.
- Final matched production timings use baseline `aa58fa1073`
(`index-BaRiMbxd.js`) and candidate UI source `f9a26b9783`
(`index-DPWTbsFF.js`, identical application source at final head
`bf71ffd5af`). Chromium 151.0.7922.34, 1440×900, 4× CPU throttle, no
added network latency, browser HTTP cache disabled, fresh application
cache for each cold visit. Both variants block service workers and use
the same warm local API server and immutable 600-message/300-run
fixture. Only document HTML is intercepted for both variants; API
requests are real. The common API revision is `109981f8c1`; subsequent
commits change only UI and tests.

| Usable conversation | Baseline median (range) | Candidate median
(range) |
| --- | --- | --- |
| Cold direct URL | 6.57 s (5.69–7.45) | 3.07 s (2.50–3.28) |
| Warm in-app return | 3.76 s (3.54–4.76) | 0.67 s (0.58–1.46) |
| Back return | 2.53 s (2.12–4.20) | 0.51 s (0.47–1.18) |

Five pairs alternate order. Every paired difference favors the candidate
for all three journeys. Readiness requires a revealed, fully visible
latest reply above the composer and an editable composer. Return timing
starts after the Tasks route renders. Cold API requests fall from
953–954 to 71; API wire bytes fall from 517–559 KB to about 49 KB.
Separate CPU profiles and request waterfalls confirm the removal of
lifetime event hydration and are excluded from the five timing pairs.
Both variants have the same expected missing optional plan document
(404) and no browser exceptions. Control recent-window API reads have
medians of 37 ms / 32 ms, with shared-host outliers up to 376 ms.

The 6,000-message recent-window SQL uses
`issue_comments_company_issue_created_at_idx` (22 index rows read for a
21-row probe). Geometry assertions retain a 2-pixel tolerance for the
reading anchor and first-reveal layout. These are local synthetic
measurements, not deployed staging latency. Earlier timing probes are
excluded because they did not isolate service workers or composer
clearance. Final head `bf71ffd5afb5684ba9105a3127f617e7adf5139c` has 55
successful checks, three optional skips and one neutral result, with no
failures or pending checks. Greptile is 5/5 with no open review
findings.

### Failure record

Early local attempts exposed an unconditional scroller wrapper that
changed the old task layout, and automatic retries on chat page
failures. Both were corrected. The original task scroller structure
stays in place. Metadata request assertions wait for initial connection
recovery to settle, then verify only the new page’s reads;
retry/resume/reconnect revalidation remains intentional.

The initial review caught a send-before-scroll cache defect; its
controller regression failed before the fix. Later rapid-paging checks
caught scroll readiness being gated by metadata after the first reveal;
readiness now stays enabled once revealed. Live replies use the separate
acknowledged-message cache, then catch up from the server cursor.

Visual review exposed a pagination wrapper bypassing existing composer
clearance. The body is again a direct scroller child; cold-open browser
checks require the full latest reply above the composer. A
resize-observer regression failed before the fix and now passes.
Controlled delayed inline images prove both pinned following and a
stable reading anchor on desktop/mobile.

The first performance fixture lacked valid native event envelopes and
caused missing-log fallbacks. It is excluded from latency claims. The
final fixture uses saved native assistant events and conversation
session generations. The live test uses a temporary fixture-agent key,
attributes the reply to an existing synthetic run, and revokes the key.
It starts no model or agent process.

Local database attempts failed during a Docker shutdown. The retained
disposable database was restored and its suite passed. Early fixture
errors included a missing conversation identity field and session
generation. Mobile test setup also selected an offscreen sidebar link
and reused queued human input. Optional announcements could intercept
input; the fixture now dismisses them before the journeys. The scroll
observer now waits for the delivered wheel event and settled animation
frames, including when the viewport is already at the top. A
concurrently running live test briefly changed another probe’s row
count; the final acceptance and timing runs were sequential on separate
fixtures. The final rig uses the visible mobile navigation and a saved
agent reply. Anchor observation waits for the user's scroll to settle
before measuring the effect of live updates.

The full local `pnpm test:run` attempt hit repeated embedded PostgreSQL
initialization failures and failures in untouched native-session
executor suites; it was stopped after the repeated environment failures.
The targeted real-PostgreSQL suite passed, and final-head CI is required
before handoff. The first CI run also caught an exact mock expectation
missing the added company scope; the expectation is updated and its
179-test suite passes. The final-head first CI attempt also stalled an
assigned child task in the existing plan-approval scenario, then hit a
heartbeat-run row lock timeout during fixture cleanup. The failed shard
and aggregate check passed on an unchanged rerun of the same commit.
Scenario assertions and timeouts were unchanged. The failure and rerun
are retained in [CI run
41116120430](https://github.com/paperclipai/paperclip/actions/runs/41116120430).

## Risks

- Historical transcript diagnostics no longer load on a cold chat open.
Saved messages and conversation events remain available; diagnostic run
views keep their full records.
- Cursor paging must not omit comments with equal timestamps. Database
tests cover timestamp/UUID ordering and the complete set.
- Prepended or late content can move the reader. Browser checks cover
retained row identity, viewport offsets, mounted drafts, and Back.
Previously loaded pages stay in the query cache for its existing
lifetime.
- Local synthetic evidence does not establish deployed staging latency.
This PR does not merge or deploy the change.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, tool use, and code
execution. The exact backend deployment ID and context window were not
exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (affected checks; limits
are recorded above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-30 15:12:24 -05:00
DottaandPaperclip 9e1735ec17 test(evals): add a fixed TestOps AWS chat feature route (#18322) 
## Thinking Path

> - Paperclip lets people manage AI agents and inspect their saved work.
> - Chat feature evaluations test a real request against saved tasks,
audit records, links and reload behavior.
> - The current suite has Codex and OpenCode cells but no dedicated
runnable DeepSeek 1017 backlog plan in TestOps.
> - TestOps must select the real case before the workflow can access a
model key.
> - This PR adds one fixed, default-branch AWS route for native OpenCode
and DeepSeek 1017.
> - The route keeps original evidence private and sends only a verified
result and approved screenshots to TestOps.

## Linked Issues or Issue Description

**Subsystem affected**

Product E2E evaluation definitions and GitHub Actions delivery.

**Problem or motivation**

A dedicated TestOps plan cannot run the existing backlog chat journey
with native OpenCode. A result must prove saved work and cleanup rather
than report that a model call completed.

**Proposed solution**

Add one OpenCode backlog cell and one fixed public workflow. TestOps
selection validation runs in a token-isolated job before builds or
inference. The workflow retains private originals and reports the bound
outcome to the selected TestOps case.

**Alternatives considered**

A private-repository wrapper would need a new cross-repository GitHub
write credential. The new route uses the existing TestOps GitHub
integration instead. Existing manual candidate approval paths remain
separate.

**Roadmap alignment**

This extends the existing feature evaluation infrastructure. It changes
no production feature, global runtime default or model default.

Current main #18319 supplies the v2 grader, delegation queue accounting
and final screenshot capture; these improvements are preserved. Related
product work: #16814 and #17126 concern backlog handling. This PR
changes neither product implementation and does not claim those changes
are required.

## What Changed

- Add exactly one native OpenCode backlog cell with DeepSeek 1017 under
the separate `runner-opencode-deepseek-1017` profile. Preserve all four
existing Codex and production OpenCode cells and their models.
- Add a default-branch-only workflow with fixed evaluation, zero-result
registration and one plan-launch modes. Require GitHub actor checks and
authenticated singleton selection before inference.
- Add a separate shipped-main evidence authority. Reuse the existing
strict private storage, scanners, inventories and scoped AWS sessions.
- Bind sanitized results, generated JUnit and producer-approved
screenshots to the exact source, run, attempt and case. Keep failed
grades and cleanup failures failed.
- Add TestOps API, CLI and result adapters with pure controls and normal
unit-test wiring. Keep the reporting token out of the evaluated process.
- Verify the current server case result and attachment metadata after
upload, without masking an earlier failed upload or evaluation.
- Apply only the existing reviewed lock correction, verify unchanged
tracked dependency inputs and bind the lock receipt across the
compiled-output handoff before frozen installation.
- Bind the initial administrator launch to one approved workflow run
number and live GitHub source/actor metadata before TestOps
authentication. Use bounded REST case reconciliation and only the
reviewed existing GitHub integration.
- Make two existing CI tests observe their intended state
deterministically: a fake clock for copy-feedback lifetime and a
completed-pass barrier for retry-cancellation batches. Production
behavior and the original assertions remain unchanged.

The two test-only fixes are intentionally included because their
observation boundaries blocked this delivery’s required normal CI. They
preserve every original assertion and change no production behavior. The
original failed runs remain recorded, and the exact causes of those
observations remain unknown.

## Verification

- 157 focused profile, catalog, oracle, filesystem admission, export,
private-path and workflow controls passed on the current-main
composition. The existing stock OpenCode models and the new exact
DeepSeek 1017 selection are checked separately.
- 44 API, CLI, filesystem and post-upload readback controls passed,
including foreign selection, changed hashes, unsafe files, failed
outcomes, expired readback and no-resend behavior.
- The new exact private session and trace layout controls produced four
genuine parent failures and pass with the selected-path extension. The
frozen 244-case candidate approval registry remains unchanged.
- The lock guard has seven filesystem controls plus seven existing
canonical-owner controls. The unrestricted old workflow failed its
actual policy calibration; the corrected frozen-install route passed.
- Eval-owning TypeScript checks, workflow lint and Git whitespace checks
passed. The separate UI test correction was checked with the focused
jsdom unit suite. The server test barrier has pure wrapper and real
scheduler controls; its DB-backed route was not run locally. No
additional local typecheck/build was run for these test-only changes.
- Full native builds, provider execution, app/database behavior and
TestOps registration are reserved for GitHub Actions/AWS. They have not
run locally.
- The initial normal CI run failed the new-workflow policy allowlist and
a heartbeat scheduling assertion. The policy fix has an actual parent
red/current green control and rejects five unsafe route variants.
Heartbeat source is unchanged and its failure cause remains unknown. The
original run is retained.
- The launch-fence actual caller fails against the published parent and
passes the five own-run contradiction controls before authentication.
- The prior `a5aa359d` normal CI completed successfully. The complete
`f9e1f289` run passed all leaf jobs except an existing copy-feedback UI
assertion, plus its dependent verification gate. The four owning UI
files were unchanged from the earlier passing head and actual tested
merge; the exact scheduling cause remains unknown.
- The correction changes only that test to actual React `act` and a
scoped fake clock. All eight component tests pass; early-hide and
never-clear source variants each fail the intended lifetime assertion.
Production clipboard/card behavior is unchanged.
- The `2f27f3fc` run passed the corrected UI shard and every leaf job
except the existing retry-cancellation intermediate-count assertion,
plus its dependent verification gate. The seven relevant owners match
the earlier passing run and historical source comparison; the exact CI
scheduling cause remains unknown.
- The test-only barrier observes a persisted completed worker pass
before asserting the unchanged 20-item cap and two-company fairness,
then releases it for the original 29-item drain and 180-second idle
check. Two extracted-wrapper controls and three actual scheduler
controls pass. The real DB-backed test awaits AWS qualification.
- On `836e80ac`, both repaired test shards passed in AWS. The original
attempt retained a separate E2E cleanup HTTP 500 from a PostgreSQL lock
timeout and one runner shutdown. No production repair or precise
contention/shutdown cause is claimed.
- Exactly one targeted retry of E2E shard 3 and its aggregate gate
passed in attempt 2; exactly one recovery of the interrupted server
shard and its verification gate passed in attempt 3. Other passed jobs
were carried forward. The final normal run is successful. No
whole-workflow retry or evaluation/model retry was launched.
- Both literal required checks (`ci / verify` and `ci / e2e`) are
successful on the exact final source `836e80ac`. Greptile is 5/5 and no
review threads remain open. Actual TestOps registration and the
OpenCode/DeepSeek feature result remain unqualified until their separate
executions.

## Risks

- Actual zero-result registration, tenant API shapes and
public-repository dispatch access must be checked before the first
evaluation. Missing or ambiguous metadata stops the route.
- The real OpenCode/DeepSeek backlog result and Allure outcome remain
unqualified until that run. No passing result is seeded.
- One human query, one accounted work run, 120-second turn, 420-second
attempt, 300-second settled-state bound and zero retries remain
unchanged. Setup usage is outside work accounting and stays unknown.
- The reporting token was configured through the separately reviewed
protected credential bootstrap. It is unavailable to AWS evaluation
jobs.
- Metadata reconciliation is capped at ten pages, 1,000 cases and four
parallel detail reads. Missing or inconsistent metadata stops before
inference. The initial administrator launch counter must be created once
after registration; it does not restrict normal TestOps Run actions.
- Existing manual workflows, historical evaluation grades and candidate
approval records remain unchanged. The original CI failures are
preserved; the test corrections do not establish retrospective causes.

## Model Used

OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) with reasoning and tool use. The
runtime did not expose its context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-30 15:08:16 -05:00
DottaandPaperclip 8f822bfcf7 fix(cloud): persist configuration and load release-image packages (#18335) 
Land the approved Cloud configuration and release-image package stack.

Includes the separately reviewed adapter settings (#18335), external adapters (#18333), bundled plugins (#18336), startup configuration (#18339), and migration/package checks (#18341).

Includes the final image-owned loader and linked-dependency inventory corrections.

Verified stack head: 462716170d.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-30 19:21:33 +00:00
DottaandPaperclip aa58fa1073 perf(ui): restore fast Coordinator task and inbox loading (#18323) 
## Thinking Path

> - Paperclip lets people manage work done by agents.
> - Tasks, Inbox, and Needs You must stay usable as work history grows.
> - The Coordinator views bypassed some existing collection
optimizations.
> - Navigation fetched complete decision cards for a count, and Tasks
mounted every loaded row.
> - This PR uses the canonical attention rules for small count and
task-ID responses. It also restores bounded rendering and the optimized
Inbox selection.
> - The benefit is less data and less initial rendering work without a
second definition of pending work.

## Linked Issues or Issue Description

**What happened?**

Coordinator navigation fetched the full attention feed on every cold
collection visit. Tasks rendered the entire loaded collection.
Coordinator could also select the old Inbox loader when Streamlined UI
was off.

**Expected behavior**

Fetch only the projection needed by each surface. Start with a bounded
set of rows. Keep complete task membership, accurate counts, current
permissions, and stable loaded rows.

**Steps to reproduce**

1. Enable Coordinator and open Tasks or Inbox with an empty browser
query cache.
2. Observe a complete attention-card download for the navigation badge.
3. Open a large task list and observe that every loaded row mounts at
once.
4. Disable Streamlined UI while keeping Coordinator enabled, then open
Mine and observe three overlapping compact task reads.

Related work: #17756, #3744, #17983, and #18312.

## What Changed

- Add board-only attention count and complete task-ID projections. Reuse
canonical company, privacy, addressee, deduplication, dismissal, snooze,
and archive rules. Skip card bodies, plan/image reads, and presentation
roster lookup.
- Share the small count read across desktop and mobile navigation. Reuse
the page-owned response on Needs You and Needs attention. Keep the
newest known count through navigation, including zero.
- Use the task-ID index for Needs attention and keep task row hydration
in batches of at most 100.
- Render 100 task rows initially, across the whole expanded tree. Reveal
150 more at a time through scrolling or a button. Keep existing rows
through refresh and append. Defer offscreen leaf layout with shared
tokens.
- Keep Coordinator on the optimized Inbox when Streamlined UI is off.
Use index reads for expanded project recents.
- Add real-database, component, and production-build browser regression
checks. Update the access and feature maps.

## Verification

- All required CI checks pass on
`1c68ac8957490a6c8585ba0939355f536c8120bf`: typecheck, build,
server/workspace tests, serialized database suites, and eight E2E
shards. Greptile is **5/5**, with its row-retention finding fixed and
the review thread resolved. Storybook checks were intentionally skipped;
Paperclip Review completed neutral.
- Workspace typecheck and production build passed locally. The final
rebased source also passed UI typecheck and production build. Token
gates and map checks passed.
- The attention service suite passed all 34 tests on disposable
PostgreSQL. Final targeted checks cover compact projections,
complete-set counts, quality approvals, snoozes, private tasks, and
board/company access. Database response capture confirms compact
projections do not read plans or images. All 31 OpenAPI tests passed
after registering both endpoint contracts.
- Six production-build browser journeys passed: desktop, mobile,
Coordinator Inbox with Streamlined UI off, Needs You, Needs attention,
and an older server page followed by task open/Back. They verify the
initial 100-row budget, scrolling, retained DOM identity/height, one
owner per attention read, complete task membership, 220 retained rows on
Back, and less than 2 px of return-position difference.
- The row-budget and Inbox regressions fail against original source and
pass with this change. The final three affected UI suites passed 40
tests, and the broader Inbox/IssuesList suites passed 101 tests
serially.

### Controlled production-build comparison

Baseline `b31ec0a2dc079e32272a8f618425fe92b68bbaec` versus PR head
`1c68ac8957490a6c8585ba0939355f536c8120bf`. Isolated baseline asset
`index-CS9uPxMx.js`; candidate asset `index-BxJeTQ30.js`. Both UI builds
used the same warm disposable API server and database, with 220 tasks
and 110 pending board approvals. Chromium **151.0.7922.34**, 1440 × 900
viewport, **4× CPU throttle**, loopback with no added network latency,
alternating baseline/candidate order. Cold cases use a new browser
context with network cache disabled; warm cases navigate within the same
context. Other host workloads are uncontrolled.

Readiness requires the destination collection's first visible task row,
then two animation frames. Inbox readiness waits for its canonical Mine
route so previous-page rows cannot satisfy the measurement. Five paired
samples per case; cold Inbox has a second five-pair confirmation wave
because the first result was slower. Times below are **milliseconds**;
parentheses show min–max. Paired savings are the median of individual
before-minus-after pairs, not the difference between medians.

| Journey | Pairs | Baseline median (range) | Candidate median (range) |
Median paired saving | Faster pairs |
| --- | ---: | ---: | ---: | ---: | ---: |
| Cold Tasks | 5 | 2752 (2542–3025) | 2300 (2275–2524) | 328 | 5/5 |
| Cold Inbox | 10 | 2536 (2283–2927) | 2582 (2117–2856) | 37 | 5/10 |
| Warm Tasks navigation | 5 | 2054 (1826–2479) | 1448 (1419–1548) | 635
| 5/5 |
| Warm Inbox navigation | 5 | 2020 (1485–2254) | 1568 (1289–1848) | 350
| 5/5 |
| Task → Back to Tasks | 5 | 1747 (1609–1948) | 1327 (1238–1355) | 472 |
5/5 |

Cold Tasks, warm Tasks, warm Inbox, and Back improved in every measured
pair. **Cold Inbox latency is inconclusive:** the two waves pointed in
opposite directions, with the candidate faster in 5/10 pairs. This PR
claims less badge work there, not a proven cold Inbox latency
improvement.

The canonical badge response shrank from **200,039 to 359 decoded
bytes** (encoded HTTP transfer: **8,125 to 598 bytes**). The compact
response also skips card payloads, plans, images, and presentation-only
roster work. These are measured mechanism reductions, separate from
end-to-end latency. Needs You still owns its full card feed; Needs
attention owns its complete task-ID projection. Reproduction setup and
assertions are in `tests/perf/collections/README.md`.

<details>
<summary>Individual paired readiness measurements (ms)</summary>

| Journey | Pair | Baseline | Candidate |
| --- | ---: | ---: | ---: |
| Cold Tasks | 1 | 3025 | 2283 |
| Cold Tasks | 2 | 2542 | 2275 |
| Cold Tasks | 3 | 2752 | 2424 |
| Cold Tasks | 4 | 2783 | 2300 |
| Cold Tasks | 5 | 2641 | 2524 |
| Cold Inbox | 1 | 2927 | 2702 |
| Cold Inbox | 2 | 2422 | 2629 |
| Cold Inbox | 3 | 2409 | 2606 |
| Cold Inbox | 4 | 2621 | 2769 |
| Cold Inbox | 5 | 2539 | 2856 |
| Cold Inbox | 6 | 2283 | 2117 |
| Cold Inbox | 7 | 2466 | 2558 |
| Cold Inbox | 8 | 2674 | 2286 |
| Cold Inbox | 9 | 2696 | 2389 |
| Cold Inbox | 10 | 2533 | 2183 |
| Warm Tasks navigation | 1 | 2479 | 1423 |
| Warm Tasks navigation | 2 | 2228 | 1548 |
| Warm Tasks navigation | 3 | 2054 | 1419 |
| Warm Tasks navigation | 4 | 1826 | 1448 |
| Warm Tasks navigation | 5 | 1954 | 1518 |
| Warm Inbox navigation | 1 | 2020 | 1661 |
| Warm Inbox navigation | 2 | 2198 | 1848 |
| Warm Inbox navigation | 3 | 1733 | 1497 |
| Warm Inbox navigation | 4 | 2254 | 1568 |
| Warm Inbox navigation | 5 | 1485 | 1289 |
| Task → Back to Tasks | 1 | 1721 | 1330 |
| Task → Back to Tasks | 2 | 1747 | 1275 |
| Task → Back to Tasks | 3 | 1948 | 1355 |
| Task → Back to Tasks | 4 | 1609 | 1327 |
| Task → Back to Tasks | 5 | 1790 | 1238 |

</details>

### Failed attempts and limits

- The local full repository command hit embedded PostgreSQL startup
failures and 16 failures in the untouched native session executor suite;
it was stopped. The full local suite is not claimed green. CI on the
final revision provides the broad test result.
- A concurrent local UI run hit timing failures under host contention.
Inbox/IssuesList subsequently passed all 101 tests serially. No tests
were weakened to address these timeouts.
- CI caught two integration gaps introduced during this work: missing
OpenAPI entries for the new endpoints, and an attention fixture
incompatible with the routine filters added on master. Both were fixed;
the affected 31 OpenAPI and 40 UI tests pass.
- The final revision's first CI attempt failed two untouched tests:
scheduled-retry batch cancellation and the tool-access audience/archive
race. Both failed jobs passed when rerun without changing the revision.
Their original failed attempts remain visible in [the workflow
history](https://github.com/paperclipai/paperclip/actions/runs/41083379875).
- Browser evidence uses synthetic disposable data. This PR is not
deployed staging or production verification.

## Risks

- New attention sources must preserve canonical eligibility. Both
projections call the existing collector, so they share its permissions
and complete-set behavior.
- The count and task-ID caches must not stand in for full decision
cards. Each projection has its own key under the existing attention
invalidation prefix.
- A row budget can omit descendants if applied per parent. The tests
cover one global budget across a large expanded tree, refresh, append,
reset, and incremental loading.
- Local browser evidence uses synthetic data and is not staging
deployment proof.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, tool use, and code
execution. The exact backend model deployment ID and context window were
not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (affected suites; broad-run
limitations are listed above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-30 11:58:53 -05:00
DottaandPaperclip 5c807d11fa fix(github): prevent duplicate review score headings (#18321) 
Clarify the shared GitHub review submission contract for new and existing bots.
Remove copied score headings while preserving the structured score and saved assessment.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-30 10:19:31 -05:00
Nicky Leach fba0ab8162 fix(runner): preserve GitHub helpers and Codex probe policy (#18212) 2026-10-30 07:42:10 -07:00
DottaandPaperclip 6da112ab40 perf: avoid flushing separate Git cache seed archives (#18308) 
## Thinking Path

> - Paperclip manages work done by AI agents.
> - Native sandbox runs prepare a workspace before the agent can
respond.
> - A shared seed cache avoids repeated Git copies and archive
preparation.
> - Cache publication also flushes a branch-neutral Git seed that is
separate from recovery archives.
> - This cache can be rebuilt. Each reader checks the archive bytes
before reuse.
> - This change omits that separate cache file flush and measures the
remaining preparation gaps.
> - The benefit is less work before agent execution and clearer timing
evidence.

## Linked Issues or Issue Description

Refs #18240.

**What existing behavior does this improve?**

Workspace seed cache publication and latency diagnostics for native
sandbox preparation.

**Current behavior**

Publication flushes both cache archives to disk under the company lock.
Slow storage can delay the publishing run and cache readers. A cache hit
also combines private copying, hashing, and last-used bookkeeping in one
span. Early controller preparation has gaps before environment
selection.

**Proposed behavior**

Keep the existing flush for recovery-backed workspace and Git archives.
Omit the flush only for a separate branch-neutral Git cache archive.
Reject an alternate cache file that is the recovery file or its
hardlink. Keep archive checksums, private copies, atomic publication,
quotas, and company locking. Measure cache read steps and the existing
early preparation operations.

## What Changed

- Skip the flush only for the separate branch-neutral Git seed used by
the company cache.
- Preserve the existing flush for recovery-backed archives, including
hardlink storage. Reject cache-only files that alias recovery bytes
before or during publication.
- Add a regression with a blocked cache-only Git flush. The deployed
code waits; the new code publishes verified cache bytes without waiting.
Test storage failure for both recovery-backed archives and same-path,
hardlink, and replacement alias cases.
- Add cache copy, hash, and bookkeeping spans beneath seed lookup.
- Add timing around existing agent, conversation, runtime, issue,
identity, trust, and task context preparation. Keep their execution
order.
- Update the trust-order guard to recognize the timed call while
continuing to assert trust resolution precedes workspace selection and
credential setup.
- Document the cache contract and the new spans. Keep the old flush span
in the allowlist for compatibility with older observations.

## Verification

- The blocked-flush regression fails on the old code and passes with
this change.
- Focused tests (`vitest run` on the six affected cache, timing, and
native-sync suites) cover cache corruption, private-copy integrity,
quota and eviction races, branch binding, per-run recovery, native
workspace synchronization, trace parentage, and tracing disabled or
failing.
- 115 focused tests and both typechecks pass after rebase on current
master. The tests include recovery storage failures and an archive
replaced with a recovery hardlink during verification.
- All seven trust tests pass with the actual service and migrations,
using a fresh database on an existing local PostgreSQL server. The local
embedded server could not allocate host semaphores, so only its test
startup helper was substituted; assertions and service code were
unchanged.
- A real SDK export with a retained public seed verifies that cache-read
detail spans have the lookup as their parent. The endpoint-off control
produces no spans.
- Current revision `28dea63432d462cad160906f8dfc2d21c81d3446`: all 53
checks pass, two optional checks are skipped, Greptile is 5/5, and no
review threads are open.
- No new staging latency claim is made by this PR. Hosted measurement
remains necessary after delivery.

## Risks

A crash or power loss can discard the cache-only Git seed. A later read
rejects missing or corrupt cache bytes and prepares the workspace again.
Recovery-backed archives retain their previous flush behavior. The
change adds no flush to warm per-run persistence. Hashes and
private-copy checks remain required. The new spans use the existing
opt-in exporter and closed names; they record no task content.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, tool use, and code
execution. The session does not expose the exact model SKU or
context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-30 09:13:08 -05:00
DottaandPaperclip 7df1656dda fix(server): continue retention pages after overdue grace deadlines (#18270) 
## Thinking Path

> - Paperclip manages AI agents and their shared work.
> - Skill snapshots need safe cleanup after their retention grace
period.
> - The retention worker scans large directories in pages.
> - Another company's grace deadline can expire while a page is still
running.
> - The shared deadline helper then uses the worker's 30-second failure
retry for successful continuation.
> - This pull request gives this worker a one-second continuation delay.
> - Directory scans keep moving while failure retries retain their
existing delay.

## Linked Issues or Issue Description

Refs #18266. Its unrelated fixture checks exposed this scheduling bug.

**What happened?**

A successful directory page can wait 30 seconds before the next page
when another company's grace deadline becomes overdue during the scan.

**Expected behavior**

Continue successful paging within one second. Process the overdue
company. Stop scheduling when all work settles. Keep the existing
30-second failure retry.

**Steps to reproduce**

Hold the second successful page with a deferred promise. Advance the
fake clock past another company's saved five-second deadline. Release
the page. Before this fix, the next wake is still 25 seconds away. The
regression also requires all pages, the overdue rescan, and an idle
worker after settlement.

**Paperclip version or commit**

Reproduced on `b3c2172d417ea6b783e0c796aaed4a91e1f9d381`.

**Deployment mode**

Core retention worker. The deterministic test uses the real coordinator,
deadline helper, and retention worker with controlled service results.
It needs no external database or provider.

## What Changed

- Set `continuationMs: 1000` on the skill retention worker.
- Add a controlled regression for a grace deadline that expires during a
successful page.
- Keep the shared deadline helper and the worker's failure retry
unchanged.

## Verification

- Red: the new regression fails on unchanged production code with a
25,000 ms remaining delay. The existing control passes.
- Green: retention worker, deadline worker, and scoped deadline worker
suites pass: 7 tests.
- `pnpm build` and `pnpm -r typecheck` pass. All required checks pass on
the current head. Greptile reports 5/5 with no review threads.
- The full local test suite is not complete. The bundled macOS
PostgreSQL runtime lacks `libzstd.1.dylib`, as confirmed in the existing
host diagnostic. The deterministic worker tests do not need PostgreSQL.
Full repository tests must pass in CI.
- No live Cloud, provider, model, or deployment test is selected.

## Risks

Successful overdue continuation can run sooner. The existing one-second
page interval still applies. The test requires the remaining pages, the
overdue rescan, and no more work after settlement. Failure retry remains
30 seconds. This change adds no schema, API, or product command.

## Model Used

OpenAI `gpt-6.1-sol`, ultra reasoning, through Codex. Parent runtime
metadata confirms this model. The context window is not exposed.
Terminal tools assisted source review and testing. A separate agent
reviewed the source and regression.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-29 23:50:08 -05:00
DottaandPaperclip ebed7f5237 test(evals): protect Antigravity setup and two-turn delivery (#18190) 
## Thinking Path

> - Paperclip manages agents and their work.
> - A provider integration must work through ordinary agent setup and
task controls.
> - The recorded Antigravity walkthrough exposed setup, delivery, and
continuation failures.
> - A repeatable check must verify real work and retain failed attempts.
> - This pull request converts all 47 recorded steps into one fixed
Product E2E journey.
> - Maintainers can rerun the case and inspect independent delivery,
cleanup, and accounting evidence.

## Linked Issues or Issue Description

Builds on merged #18187, #18188, and #18189, and consumes the shared
repairs from merged #18230 and the equivalent manual-policy declaration
from #18222. Refs #18214 for the CI task wrapper. Refs #12503 and #14938
for related legacy adapter work.

**Agent or provider**

Google Antigravity native Runner with a saved managed Google
subscription and the exact Low model.

**Why this adapter is useful**

A user must be able to add an agent, connect its account, and receive
real task results across a follow-up. Picker visibility alone is
insufficient.

**How the agent is invoked**

A fixed Playwright driver uses ordinary product controls. The agent runs
through native ACP. Independent checks use public APIs, actual
downloads, owned process observations, and native lifetime fences.

## What Changed

- Register one explicit-only case:
`antigravity-agent-journey.runner-antigravity.local.saved-subscription-two-turn`.
- Map all 47 source steps and declare saved-account prerequisites,
finite limits, and same-host macOS ARM64 scope.
- Create an agent through the ordinary picker, connect the saved
account, select Low, run Test, and wait for readiness. Save an
instruction marker, deliver a file, and update it through an ordinary
follow-up.
- Require each run's visible completed file activity, exact downloaded
bytes, committed Done result, same tagged conversation, distinct
physical owner birth, released lease, and free native fences. Verify
reload persistence.
- Stop failed native runs that still own a retry. Replay an existing
Stop with its durable request identity and wait for the matching audited
acknowledgement. A terminal reservation without a native intent remains
unconfirmed.
- Keep private evidence in the existing result, report, dashboard, and
accounting pipeline. Retain the paused QA agent and financial history
while preserving the shared account and budget policies.
- Journal only the ordinary wizard's future ID, target, name, intended
harness and expected creator before forwarding its POST unchanged. Bind
the response and interrupted recovery to exact identity reads; a pending
404 stays unconfirmed.
- Serialize private resource snapshots through temporary-file fsync,
atomic rename and directory fsync. Abort forwarding on storage failure
and preserve legacy receipts. If later journal or diagnostic writes
fail, retain the primary error, always close the browser, and attempt
the ordinary failed report.
- Add production-component browser controls and harden task-menu
readiness, alias selection, scroll restoration, failure packaging, and
one hermetic auth-test budget.
- Wait for the existing GitHub recovery fixture to register its next
deadline before asserting the original range; preserve its
nested-transaction and zero-request checks. In the decision-expiry
fixture, verify the exact saved expiry separately from the earlier
global wake of another queue.

## Verification

- Fresh registered acceptance passed on eval source
`9060f3cd01b7c3445ecdf703d72e9cf30fe42b59` and merged serving product
`61bd4ac6919c2309f766a25e62273b074cee3261`. It completed ordinary setup
and two turns with native `.5` and `gemini-3.8-flash-low`. All 16
matchers, 14 checkpoints, and cleanup passed in **165,613 ms**. Actual
downloads were **61 bytes** and **119 bytes**. Both turns passed their
own run-bound browser activity checks and reload persistence.
- Independent read-back confirms Done, a paused retained agent, the same
connected saved default account, and no active owned run. Both accepted
results have the same full tagged conversation and distinct primary
process births. All 21 observed processes retired; both leases released
and all three native fences were free after each turn. The selected `.5`
checkpoints also showed equal agent/backend session IDs; this is an
observed case requirement, not a generic ACP invariant.
- Receipts report 339,273 input tokens, 7,224 output tokens, and 202,846
cached input tokens. Subscription prices are unavailable. Billing
coverage remains incomplete, not free.
- The normal report CLI passed. All 310 files from nine original
attempts and two separate recoveries retained their hashes. The twelve
current screenshots are byte-identical, and 194 local report links are
contained and valid. The eight earlier reports were copied without
regeneration.
- Delivery source `1bf4fb04df04710b82844b3c9355767f46b5f68f` branches
from `d3f2a60e795fc60d64c8b19d1eed4d48657d5492` and merges cleanly with
current master `1d571de5f9f886d66e520800baec8fc2f67d0948`. All 41 prior
`c8b332` authored file blobs remain identical; the 42nd path is the
decision-expiry fixture contract correction. Creation ownership, the
strict durable writer, and the ordinary successful workflow remain
unchanged. Of 43 measured file blobs, 30 remain identical. The
Antigravity oracle, artifact, billing and report gates and its
suite-specific validation block remain unchanged. The actual retained
agent matches its original creator; its original request `setupId` was
not captured and is not reconstructed.
- Later master changes include model-visible workspace guidance and
rejected-terminal-report handling. Their bounded source assessment is
separate: the first actual finish failed, then a same-turn replacement
was accepted and committed. Later read-only results match both original
native-result hashes, and the complete event catalogs have no
`session.failed` event. Master #18228 also expands detection of explicit
JSON file requests. The measured create/update-and-attach `.txt` task
already required publication under both parsers; the publication
validation gate itself is unchanged. Master #18240 changes remote
workspace cache staging and adds an OpenCode index refresh. The selected
local lane returns before those paths. Master #18213 adds optional
numeric turn timing and trace boundaries. Its selected Antigravity
model, text, IDs, result and retirement semantics remain unchanged in
the bounded source review. Local diagnostic rows and awaits can change
event ordinals and latency; the oracle checks relative semantic order.
Collector coverage and latency improvement remain outside this proof.
This does not claim that the whole current runtime is byte-identical to
serving `61bd`, or relabel the measured result as a new run.
- The adopted upstream cache controls passed 3/3 with pinned real Turbo.
Forced work disables local and remote reads and writes; ordinary work
reuses local results. The original genuine-pnpm/Turbo controls remain
historical to `9060`'s earlier write-only policy. The duplicate cache
patch is superseded by #18230. Its qualification-only pnpm addition is
removed because the adopted upstream controls do not require it.
- The E2E typecheck, map checks and 277-control replay are carried from
the unchanged eval source paths. That replay passed, including the 183
Antigravity controls, catalog/connection checks and the separate
product-journey validator. The carried creation controls include 91
config/recovery cases. The final launcher suite passed 19/19, including
five real campaign finalization fault cases; the report suite passed
7/7. These final checks did not rerun the earlier 277-control group. The
canonical production-component browser spec passed in 27.4 s under its
unchanged 60 s limit before the replay. Four paused-composer cases and
the product base's build/typecheck plus 376 focused regressions were
verified earlier.
- The shipped wizard and actual API client cover persistence before
forwarding, lost response JSON, foreign/malformed requests, wrong
response identity/creator and cleanup-error preservation. Real
filesystem controls failed 5/5 before the writer repair and passed 5/5
afterward, preserving the old journal on partial-write, temporary-file
sync and rename faults. Three persistent-sync campaign controls failed
before the finalization repair and passed afterward. All five final
campaign fault cases preserve the primary failure, cleanup failure,
usage and failed report while attempting browser closure. The strict
journal still blocks the next mutation. These are provider-free
ownership controls; the Google task proof remains the original `9060` →
`61bd` acceptance.
- The decision restart check passed in the local 20-case fixture run. A
current-source real-PostgreSQL rollback control reproduced the exact
90,000 ms earlier global wake: the old equality rejected it, while the
exact durable expiry and new future-wake bound passed. The original 60 s
open-state, exact-timestamp, continuation and quiet checks remain. The
broader local run was **FAILED: 19 passed / 1 failed**, with a 15 s
timeout and a 30 s cleanup timeout in an unchanged timestamp-pagination
case. That case passed in canonical `c8b332` CI; the local timeout cause
remains inconclusive. An extra local server typecheck failed against
stale cached Runner declarations; it did not check this excluded test
file. Both limits remain recorded for fresh canonical CI.
- A broader local GitHub recovery group passed 21/22. Its unchanged
pagination case observed an unexpected config request; the log does not
establish the cause. That result remains retained, and fresh
current-head CI is the final broader gate.

- Exact delivery-head CI completed with 55 successes, two skips, and one
optional neutral review. Required `ci / verify` and `ci / e2e`, build,
typecheck, and Runner checks passed. Fresh Greptile is 5/5 with zero
unresolved threads. Optional Paperclip is neutral 4/5; its carried
recommendations are dispositioned against the public Stop contract and
selected live identity evidence.

Seven earlier converted attempts retain their failed grades and partial
evidence. Historical Paid8 and its later read-only per-run UI
observation remain separate from the fresh acceptance above. Unknown
prices and the older partial startup accounting remain explicit. Full
lineage and raw evidence remain private in the normal report bundle.

## Risks

- This case requires an authorized saved personal default Google
subscription and finite operator candidate admission. It does not test
fresh OAuth or establish global qualification.
- The physical observer supports a same-host loopback macOS ARM64
target. Cloud, Linux, and other models remain outside this case.
- A known-cost stop cannot cap missing prices or in-flight subscription
charges. Existing operator policies remain unchanged.
- Raw account/native identities stay private. Missing retirement proof
keeps resources retained. The selected cleanup mode does not test agent
purge.

## Model Used

OpenAI Codex (GPT-6). The host did not expose the exact model revision
or context limit. The model used tools for code edits, tests, browser
automation, and review.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-29 19:51:32 -05:00
Nicky LeachandPaperclip 1d571de5f9 Add shared native turn attribution boundaries (#18213) 
## Thinking Path

> - Paperclip manages work performed by AI agents.
> - Native execution crosses the plugin worker, Runner, sidecar, and
durable event boundaries.
> - Existing outer spans leave repeated preparation and admission work
unattributed.
> - Optimizations need a common baseline with accurate ownership and
event meanings.
> - This change extends the existing opt-in tracing contract with
bounded timing receipts.
> - It preserves execution policy and keeps provider internals opaque
where the API provides no timing proof.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Attribution of native task preparation, retained/cold Runner handoff,
and completion-to-text work.

**Current behavior**
Worker spans use a flat provider contract. Workspace calls lack a
consistent dispatch envelope. Submitted-to-started timing is named as a
provider queue even though it includes other work. The transport does
not expose a shared breakdown of its waits.

**Proposed improvement**
Add invocation-scoped nesting, bounded numeric remote receipts,
current-run transport timing, durable receipt intervals, and distinct
completion/text milestones. Reuse the existing exporter. No latency
optimization is included.

Related public work: #17842 consolidates existing startup tracing tests;
it does not add this runtime coverage.

## What Changed

- Extend the optional SDK tracer with nested async scopes and completed
timing receipts. Keep the host responsible for parent identity,
capability checks, attributes, clocks, depth, and count limits.
- Add dispatch and bounded host-read timings for environment and
workspace providers.
- Resolve transport parents on each call so retained sessions do not
retain an earlier run's trace.
- Split cold/resume/attach/prepare/registration/command/started-event
waits. Add opt-in numeric ACPX Runner/sidecar status and prompt-start
receipts.
- Distinguish first completion request, validated tool receipt, final
result commit, first nonempty text, and event persistence. Rename the
submitted-to-started interval to match its meaning.
- Link completed earlier remote lifecycles to their host-owned observer
rather than placing them under a later polling request. Preserve
sampling, timestamp limits and budgets.
- Document the versioned coverage map, causal clock limits, and baseline
acceptance requirements.

## Verification

- Current head: GitHub CI is fully green. Standard Greptile and Apex are
both 5/5, with no unresolved review threads. Full-suite validation was
performed in CI; follow-up local validation was limited to affected
packages and tests.

- Review follow-up: early rejection, missing provider capability and
startup-parent regressions were reproduced and fixed. The affected
worker/SDK/timing set passes 128 tests; 28 timing/context tests, SDK
typecheck and direct server TypeScript checking pass. The server
typecheck wrapper reached an unrelated Runner rebuild and stopped
because Cargo was absent from PATH; full Runner/image validation remains
in CI.

- CI failure accounting: the earlier Turbo cache/force conflict and
Linux OpenCode shell assertion were reproduced and diagnosed. A direct
proc probe ruled out permissions; a portable shell comparison identified
Bash skipping BASH_ENV with stripped SHLVL and socket stdin. Upstream
#18230 now supplies the CI and ignored-stdin fixture fixes; this PR
adopts them without duplicate CI changes. The strict Linux OOM assertion
passed before this source-equivalent merge.
- Retry follow-up: the three insertion/replay/failure timing cases and
server typecheck pass. The existing database regression now asserts a
single marker across identical result replay; the current-head CI
integration gate passes.
- Upstream CI alignment: all three real-Turbo 2.11.10 integration cases
pass. The repository policy gate rejected committing a generated
lockfile; the final PR contains no lockfile edit and uses the maintained
inline stale-lockfile recovery.
- Linked-lifecycle follow-up: 44 focused host/export tests pass (three
optional SDK-dependent cases skip); SDK and server typechecks pass. The
API contract test verifies an explicit root context, host-owned observer
link and unsampled suppression.
- SDK typecheck and focused worker/invocation tests: 30 passed,
including clock changes, bounded legacy IPC, and manual-span
creation-scope parentage.
- Focused reviewed host/native/transaction-boundary checks: 44 passed.
The 86-test worker-manager suite passes after updating its exact
assertions for the host-owned timing budget found by CI. Earlier
preparation/first-text checks also passed.
- Affected server and Runner TypeScript checks pass after building local
workspace declarations.
- Merge follow-up: the profile/candidate startup subset passes 94 tests,
the four Grok source-identity checks pass, and Runner TypeScript
checking passes. A subsequent execution-factory merge passes 21 focused
timing/trace tests and server TypeScript checking.
- Pinned Rust build and the ACPX Rust subset pass. The hostile receipt
projection test passes. Source profile compatibility checks: 127 passed,
one existing skip. Grok identity checks: four passed.
- Broad transport/ACPX suite: 410 passed after building all fake
provider binaries. Earlier attempts failed on missing
Runner/fake-provider outputs; building the required fixtures resolved
those failures.
- The first CI run correctly rejected stale source attestations. After
integrating current master, Grok advances 13→14 and Copilot 84→85;
master’s historical fixtures and Pi 35 stay byte-identical. Provider
dependencies, models and permission defaults are unchanged. Identity
generation/checks now pass.
- The added real-Postgres result lock regression could not initialize
its local embedded cluster. The insertion/replay/failure unit
regressions pass. The current-head CI integration gate passes.
- Comprehensive repository checks and image builds are delegated to CI,
as requested for this instrumentation pass. No deployment, model run,
collector-ingestion proof, or latency improvement is claimed.

## Risks

The new SDK methods are optional. Legacy span calls and endpoint-off
behavior remain supported. The host accepts only closed timing names and
bounded attributes. Remote durations are anchored at local receipt;
transport/scheduling uncertainty remains explicit. Trace trees have
count/depth limits. The first rollout must inspect actual collector
exports before treating the baseline as accepted. Fresh live
qualification for the new source identities remains a rollout
requirement. Revert these commits to remove the new measurements; no
schema migration is required.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning and local code/tool
execution. The exact runtime model ID and context-window size are not
exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have described the issue in-PR following the enhancement
template
- [x] I have not referenced internal/instance-local Paperclip issues or
links
- [x] My branch name describes the change and contains no internal issue
details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Standard Greptile and Apex are both 5/5, with no unresolved
findings
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-29 17:45:56 -07:00
ARTeMobAgentsandARTeMobAgents 4de82bc12e test(server): give the description-edit admission poll the same retry budget as its siblings (#17487) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - One subsystem is the chat channel control plane. It admits inbound
GitHub mentions and then creates a chat conversation.
> - Admission is durable. A transient database error defers the wakeup
and retries it later.
> - The retry delay has a floor of 1000 ms (`1000 * 2 ** attemptCount`
in `server/src/services/chat-channels/inbound/wakeups.ts`).
> - One integration test waits 1000 ms for the conversation. Any
deferred retry therefore fails the test, although the product works.
> - This pull request gives that one wait the same 10 s budget that the
neighbouring tests already use.
> - The benefit is a stable `chat (2/3)` CI shard. No production code
changes.

## Linked Issues or Issue Description

No public GitHub issue exists for this test defect. This is a bug
report.

**What happened?**

The test "authorizes description edits as the editor, never inheriting
the original author's access" failed in CI at
`server/src/__tests__/chat-channels.integration.test.ts:5636`.

The assertion was `expected +0 to be 1`. It is an `expect.poll` call
without a timeout, so the budget is the 1000 ms default.

The server log from the same job shows that the application did not drop
the conversation. It shows a deferred admission instead:

```
WARN: accepted inbound chat wakeup will retry {"deliveryId":"..."}
caused by: PostgresError: could not obtain lock on row in relation "chat_endpoints"
```

`acceptInboundWakeup` sets `retryAt = now + Math.min(30_000, 1000 * 2 **
Math.min(attemptCount, 5))` on such errors. The floor is 1000 ms. One
deferred retry always exceeds the poll budget.

**Expected behavior**

The test waits long enough for one deferred retry. It then passes,
because the conversation is created.

**Steps to reproduce**

1. Check out `master`.
2. Run `cd server && ../node_modules/.bin/vitest run
src/__tests__/chat-channels.integration.test.ts -t "authorizes
description edits as the editor" --config vitest.config.ts`.
3. Add database contention on the `chat_endpoints` row during the second
`send()`, so the first admission attempt fails with `could not obtain
lock`.
4. The wakeup is deferred with a 1000 ms delay. The poll stops at 1000
ms and fails.

**Paperclip version or commit**

`master` at `991c47b`. The failure was observed on a pull request branch
that does not change the chat code.

**Deployment mode**

Local trusted, embedded Postgres test runner.

## What Changed

- Add `{ timeout: 10_000 }` to the `expect.poll` call at
`server/src/__tests__/chat-channels.integration.test.ts:5636`.
- This matches the three neighbouring tests that poll the same
conversation count (lines 5577, 5658, 5721).

## Verification

- Before the change, one deferred admission fails the assertion, because
the default budget is 1000 ms and the retry floor is 1000 ms.
- After the change, on `master` at `991c47b`:

```
cd server && ../node_modules/.bin/vitest run src/__tests__/chat-channels.integration.test.ts \
  -t "authorizes description edits as the editor" --config vitest.config.ts
→ Test Files 1 passed (1); Tests 1 passed | 1421 skipped (1422)
```

- The complete file was not re-run locally. The targeted test covers the
changed assertion. All other lines of the test stay the same.

## Risks

- Low risk. The change touches test code only.
- A longer budget can hide a slow admission, but only up to 10 s. The
neighbouring tests already accept that budget for the same behaviour.
- A real loss of the conversation still fails the test.

## Model Used

- Provider: OpenCode. Model: `deepseek-v4.1-flash-free`. Context window:
200K. Modes: tool use, code execution, file access. No extended
thinking.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: ARTeMobAgents <11891309+ARTeMobAgents@users.noreply.github.com>
 2026-10-29 18:33:15 -05:00
DottaandPaperclip 61bd4ac691 feat(agents): set up Antigravity with saved Google subscriptions (#18189) 
## Thinking Path

> - Paperclip manages agents and their work.
> - Users choose providers and managed accounts while adding agents.
> - Antigravity was missing from the ordinary additional-agent picker
and its saved Google path did not complete native readiness.
> - A user needs a successful Test, an enabled task action, and a stable
account binding.
> - This pull request connects the ordinary setup flow to the native
candidate.
> - It also preserves retired credential refreshes across database
failures and server restarts.

## Linked Issues or Issue Description

Built on merged #18188 and #18187. Refs: #12503, #14938 for related
legacy CLI proposals.

**Agent or provider**

Google Antigravity ACP 1.4.3.

**Why this adapter is useful**

Users need to set up an Antigravity agent and run real work through
their managed Google subscription.

**How the agent is invoked**

Native Paperclip Runner uses the pinned Antigravity ACP runtime. Exact
operator admission remains required; the local candidate result does not
enable general release.

## What Changed

- Show Google Antigravity in the ordinary adapter grid and preserve the
draft choice when setup resumes.
- Reuse the same user’s saved personal default Google subscription and
discover the exact low model through the native account.
- Run a prompt-free native Test and wait for authoritative readiness
before enabling task assignment.
- Carry the managed account through native runs and recover retired
refreshes through the existing encrypted, fsynced journal with exact
generation checks.
- Retain the latest rotated credential through verification, storage,
and partial home-removal failures. Retry the exact account’s retained
cleanup before another provider use.
- Reconcile cleanup before taking the credential mutation lock, fence
owners arriving during lock acquisition, and support nested lock holders
without a join cycle.
- Bound sign-in retirement outside database transactions and preserve
claims for the existing cleanup owner; one pending sign-in does not
block unrelated accounts.
- Cover Gemini and Pi API-key recovery, keep API-key bulk manual-model
editing usable, and keep adapter discovery available after candidate
expiry.
- Skip taskless native timer work before run or credential admission.
Preserve current master monitor and deadline scheduling.

## Verification

- The registered local saved-Google-subscription journey passed all 16
matchers in 196,520 ms using eval source `12aba` against actual serving
source `4a37ee7c`. Both turns stayed in one conversation, produced exact
downloaded files of 61 and 119 bytes after reload, and used distinct
native primary processes. All observed owners retired, leases released,
and credential fences settled. Independent readback confirmed Done work,
no active owned runs, a connected saved account, and the retained agent
paused for inspection.
- The measured runtime is production-equivalent to pre-restack
`65ef31dc` except its test-only transport fixture correction.
Current-master head `ff7c9249` preserves all 99 product paths: 98 exact
blobs and only eight inherited plugin-panel lines in `agents.ts`. Its
native binary was rebuilt for inherited Codex-only diagnostic changes;
the changed Rust behavior is outside Antigravity's ACPX execution path.
Shared native/UI changes have focused regression coverage. The recorded
live result is not represented as a measured replay of `ff7c9249`.
- Full workspace build and typecheck passed on `ff7c9249`. All 376
focused tests in 14 files passed, covering Google credential/retirement
controls and the inherited native-tool, redaction, route, and UI
recovery changes. Token gates, maps, and diff checks passed. The
required-close fixture was separately demonstrated red, then passed all
317 selected provider-wiring cases.
- [Current-head
CI](https://github.com/paperclipai/paperclip/actions/runs/40959387972)
is green: 55 successful checks, two skipped workflow checks, and an
optional neutral review check. Required `ci / verify` and `ci / e2e`
passed. Greptile and Paperclip Review both report 5/5 on exact
`ff7c9249`, with zero unresolved review threads.
- This qualifies the selected operator-authorized local
saved-subscription candidate. The original exploratory journey remains
historical evidence; the repeatable fixed-driver eval is in the next PR.

## Risks

- Google refresh recovery must preserve the exact grant, account, and
credential generation. An uncertain native owner or failed encrypted
handoff retains the private home and blocks cleanup.
- A successful local saved-subscription candidate does not qualify fresh
OAuth, API keys, Cloud, Linux, other models, or general release.
- Timer filtering changes only taskless native Runner admissions.
Existing task-bound monitors and legacy adapters retain their owners.

## Model Used

OpenAI Codex (GPT-6). The host did not expose the exact model revision
or context limit. The model used tools for code edits, tests, and
review.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-29 16:51:50 -05:00
DottaandPaperclip 54cc522e86 feat(runner): add gated native Antigravity ACP execution (#18188) 
## Thinking Path

> - Paperclip manages agents and their work.
> - Native Runner must bind each provider turn to the selected task,
account, model, and instructions.
> - Antigravity needs native ACP admission and physical process
ownership across continuation.
> - A saved conversation must not reuse an old process owner or silently
switch its effective model.
> - This pull request adds the native candidate behind exact operator
admission.
> - The default release gate remains closed while qualification is
pending.

## Linked Issues or Issue Description

Builds on merged #18187. Product account setup and the candidate
producer are in dependent #18189. The repeatable browser journey is in
#18190. Refs: #12503, #14938; those proposals use legacy CLI adapters.

**Agent or provider**

Google Antigravity ACP 1.4.3.

**Why this adapter is useful**

Users need to set up an Antigravity agent and run real work through
their managed Google subscription.

**How the agent is invoked**

Native Paperclip Runner uses the pinned Antigravity ACP runtime. Exact
operator admission is required while qualification remains pending.

## What Changed

- Add native Antigravity ACP model and instruction admission,
selected-account homes, and fenced refresh ownership.
- Carry exact candidate authority from the control plane through Runner,
Rust, the sidecar, and each current run. Clamp daemon lifetime to the
absolute permission expiry, including replacement owners.
- Preserve a rotated OAuth refresh token before later identity
verification can fail, including failed or cancelled startup admission.
Make private lease-close retries safe.
- Bound native probe retirement and retry the exact original owner; do
not delete credential homes before physical proof.
- Preserve Gemini API-key recovery when shared Google subscription
capability is added.
- Resume the same tagged native conversation with fresh per-turn process
ownership and physical native retirement.
- Include native manifests in isolated Rust image stages, release cache
keys, and dev rebuild detection. The workflow changes only extend
existing cache input lists.
- Keep current master provider behavior and append Grok 13, Pi 35, and
Copilot 84 input identities. Preserve prior identities, including saved
Copilot 82 and 83 runs.

## Verification

- Runner TypeScript main checks, profile generators, Rust formatting,
and offline workspace checks pass. The native backend integration target
passes 28 tests.
- The complete transport and cleanup group passes 387 tests without
unhandled rejections. These checks use local scripted children and
actual process and port ownership.
- Saved-profile decoding, native-execution parsing, installation, and
selected-credential controls pass 208 focused tests. All 109
pre-existing profile identity files remain byte-for-byte unchanged.
Copilot, Grok, and shared TypeScript/Rust generation checks pass.
- The final recovery fixture passes all seven cases. After the master
rebase, the combined recovery and owned-Node-runtime group passes all 10
cases. Each expiry case sets one 45-second absolute grant before
startup. It includes the deliberate six-second cold delay, four existing
eight-second command budgets, and active-turn headroom. The test proves
an active turn before expiry and then checks the daemon, owned process
group, child processes, and all three native port fences. The 60-second
test deadline does not renew provider authority.
- The consolidated model-discovery and pending-release group passes 22
tests. Google credential compatibility and auth recovery pass 25 tests,
including existing Gemini API-key recovery. The full product settings
test group remains intact in #18189.
- Earlier failures remain in their original CI runs and local records.
They exposed the Copilot 82 decoder gap, an expired five-second fixture
bootstrap ticket, and a fixture grant consumed before activation on
loaded CI. Each repair has a separate result. Production ticket limits
and admission behavior are unchanged by the final test commits.
- An unrelated decision-expiry check failed in [the earlier CI
run](https://github.com/paperclipai/paperclip/actions/runs/40941986698/job/123688668997).
Its owning code is unchanged from the base. The unchanged suite passes
20 tests on isolated PostgreSQL. A controlled database rollback
reproduces the earlier wake: clean startup schedules the 120-second
decision expiry, while rollback reconciliation leaves a 30-second
continuation-coalescing wake. The decision stays open, with no wake
delivery or error. The fresh final-head server shard passes; the failed
result is preserved.
- Final native commit `ec6bc7e4d77f5ae0cc4946dbba6c662e95a87f5b` passed
`ci / verify` and `ci / e2e`. Fresh Greptile and Paperclip reviews both
score 5/5, with no open findings. Antigravity retains the recorded
native `.5` closure and exact low-model profile. This PR does not claim
global or cross-platform qualification.

## Risks

- The new transport carries private managed credentials. Exact
ownership, account generation, model, instructions, and native
retirement are fail-closed.
- Only local macOS ARM64 has a verified complete Antigravity runtime.
Default Antigravity task execution remains disabled without current
trusted operator admission. The Google server sign-in lane and
production candidate producer ship in #18189; this native slice does not
expose the subscription catalog choice.
- Shared launch inputs change provider identities. New identities are
append-only; existing recorded identities remain available.

## Model Used

OpenAI Codex (GPT-6). The host did not expose the exact model revision
or context limit. The model used tools for code edits, tests, and
review.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-29 16:36:12 -05:00
Dotta 6df9fc2221 test(server): consolidate duplicated startup-tracing unit suites (#17842) 
<!-- entire-trail-link-start -->
https://entire.com/gh/paperclipai/-/trails/48/changes/paperclip/48
<!-- entire-trail-link-end -->

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The server can export manual OpenTelemetry startup spans through
`server/src/instrumentation.ts` (`resolveProtocol` plus the
`getStartupTracer` / `getStartupTraceContext` accessors).
> - Two suites covered that one module in every server Vitest run:
`server/src/instrumentation.test.ts` (unit cases added with the
manual-span seam in #10565) and
`server/src/__tests__/instrumentation.test.ts` (the bootstrap machinery
from #3735/#12249).
> - `resolveProtocol` and the startup tracer accessors were registered
twice per run with overlapping but unequal tables, so the same contracts
were executed by two files while each held a few unique rows the other
lacked.
> - This pull request moves the unique assertions into the `__tests__`
owner and retires the colocated file, so one suite exercises the full
contract once.
> - The benefit is one fewer suite registration per server run and one
less duplicated unit surface to maintain, with no lost assertion.

## Linked Issues or Issue Description

**What happened?**

The server Vitest project ran two test files against
`server/src/instrumentation.ts`. Both registered `resolveProtocol` and
startup-tracer accessor cases. The colocated file held unique coverage
(protocol input trimming, warn-once, the `getStartupTraceContext`
accessor, the `setAttribute`/`setStatus` span surface) while the
`__tests__` file held the bootstrap, peer-version, shutdown, and
span-recording coverage. The split duplicated executions and made the
contract's owner ambiguous.

**Expected behavior**

One suite (`server/src/__tests__/instrumentation.test.ts`) owns every
`server/src/instrumentation.ts` unit contract. Every assertion from the
retired file still runs, inside the owner. No production code changes.

## What Changed

- Retired `server/src/instrumentation.test.ts` after migrating its
unique assertions into `server/src/__tests__/instrumentation.test.ts`.
- Added the `OTEL_EXPORTER_OTLP_PROTOCOL=" HTTP/PROTOBUF "` row, proving
surrounding space is trimmed (previously only the letter-case row
existed).
- Strengthened the unknown-protocol case to assert the warning fires
exactly once.
- Extended the no-op tracer case to exercise `setAttribute` and
`setStatus` before `end`.
- Added the `getStartupTraceContext` accessor case (tracer surface plus
the `contextWithSpan` helper), previously covered only by the retired
file. Six production callers use this seam (heartbeat runtime execution,
native run trace, duplex observability recorder, and others).

## Verification

- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/instrumentation.test.ts` — 21 passed, 2 skipped (SDK-gated
span-recording cases skip without `@opentelemetry/*`, as before). The
two new/extended cases pass inside the owner.
- Deliberate failure controls, both restored byte-exact afterward and
re-run green:
- removing `.trim()` from `resolveProtocol` made exactly the migrated `"
HTTP/PROTOBUF "` row fail;
- making the no-op `contextWithSpan` throw made exactly the migrated
accessor case fail.
- The retired file's six remaining `resolveProtocol` rows and one
`getStartupTracer` case were exact-value duplicates of owner rows (same
inputs, same expected protocol/package pairs, same no-op tracer shape).
- Full repository typecheck/test/build were not run locally (no Rust
toolchain for the Runner build in this environment; server-wide `tsc
--noEmit` exceeds the session window). CI is the authoritative gate for
those; this diff touches only test code under `server/src`.

## Risks

Low. Test-only change; no production behavior, routing, or CI
configuration touched. Known stale mentions of the retired path are
non-functional: `.devin/wiki.json` lists it as a non-exhaustive wiki
anchor (its own instructions say to relocate removed anchors at
generation time) and `scripts/general-server-shard-durations.json` keeps
a historical duration key that no longer matches any selected file.

## Model Used

- Provider: Z.ai via OpenRouter (agent: Paperclip Test Gardener)
- Model ID: `openrouter/z-ai/glm-5.3`
- Reasoning: standard chain-of-thought mode, tool use (shell, file edit,
GitHub CLI)
- Context window: 200k tokens (provider default for this model)

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` /
`Closes: #` / `Refs #` OR (b) described the issue in-PR following the
relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes (none
required: test-only change)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
 2026-10-29 14:08:23 -07:00
Ethereal49 649354d8c4 test: wait for Slack workers before checking settled queues (#16698) 
## Thinking Path

> - Paperclip is an open source app that manages AI agents for work.
> - Slack delivery uses scheduled workers to process queued actions.
> - The queue test checks that a settled queue does not poll the
database.
> - The next scheduled wake can be empty while a worker still runs.
> - This pull request waits for the tracked workers to stop before that
check.
> - The test still rejects all queries during the 1200 ms quiet period.

## Linked Issues or Issue Description

**What happened?**

The test `wakes Slack session sync from publication commit and leaves
settled action queues quiet` can report three unexpected `select` calls.
The worker can clear its scheduled wake before its final deadline read
and empty queue scan end. The test starts its query spy in that
interval.

The failure was observed in [this CI
job](https://github.com/paperclipai/paperclip/actions/runs/38854774831/job/116823717452)
on [the SSH summary-slot pull
request](https://github.com/paperclipai/paperclip/pull/16535). This test
race also reproduces on `master` without the SSH change. This pull
request contains only the independent test fix.

**Expected behavior**

Start the quiet-period assertion after the scheduled work and the active
workers are both idle. Continue to fail if any query occurs during the
next 1200 ms.

**Steps to reproduce**

1. On the base commit below, run the target test. The failure depends on
timing.
2. To expose the interval, add this temporary control after the
coordinator is created in that test, before work is registered:

```ts
const readDeadline = f.service.nextChatActionAt.bind(f.service);
vi.spyOn(f.service, "nextChatActionAt").mockImplementation(async (kind) => {
  if (kind === "slack_session_sync") await new Promise(resolve => setTimeout(resolve, 150));
  return readDeadline(kind);
});
```

3. Run `pnpm exec vitest run
server/src/__tests__/chat-channels.integration.test.ts -t 'wakes Slack
session sync from publication commit and leaves settled action queues
quiet'`. The old wait fails with three `select` calls.
4. Apply this change and run the same control. The test passes with zero
queries during the quiet period.
5. Remove the temporary control. It is not part of this pull request.

**Paperclip version or commit**

Base: `4415665d67b16ffefbe4519c4bdd521f37bb87b9` on `master`.

**Deployment mode**

Local test run from source. The fixture uses embedded PostgreSQL and the
test Slack runtime. No live Slack workspace was used.

## What Changed

- Require both `coordinator.nextWakeAt() === null` and
`idleWorkSnapshot().active === 0` in the existing wait.
- Keep the query spy, 1200 ms observation period, and zero-query
assertion unchanged.
- Reuse the existing import and wait pattern. No production code,
dependencies, or configuration changed.

## Verification

- Controlled reproduction on the base: one target test failed with three
unexpected queries. With the same 150 ms control and this fix: one
target test passed. The control was then removed.
- `pnpm exec vitest run
server/src/__tests__/chat-channels.integration.test.ts -t 'returns two
answered Slack turns to Idle in the same thread|wakes Slack session sync
from publication commit and leaves settled action queues quiet|sleeps
behind an unresolved Slack tool send and wakes when its receipt
settles'`: three tests passed; 1344 tests were skipped by the name
filter. This run used the final source without the temporary control.
- `pnpm exec vitest run
server/src/__tests__/delivery-work-coordinator.test.ts
server/src/__tests__/work-scheduler.test.ts`: 20 tests passed in two
files.
- `git diff --check`: passed.

Environment: macOS ARM64, Node 26.12.0, pnpm 9.15.4, Vitest 5.0.3.
Existing dependencies were reused in an isolated worktree. The
shared/plugin SDK build passed.

The CI chat shard was attempted with `pnpm exec node
scripts/run-vitest-stable.mjs --mode general --group general-chat
--shard-index 2 --shard-count 3`. The runner verified its collected
filters, then reported `No test found ... in lines ...`. That invocation
skipped all 1347 tests. Its sibling invocation was stopped after this
runner failure. The shard did not pass. The first name-filtered retry
stopped in fixture setup with `ENOSPC`; task-owned scratch directories
were then removed before the final retry.

Full workspace typecheck, test, and build were not run. The isolated
environment does not have the full CLI/workspace dependencies or Cargo.
Linux with Node 24, new pull request CI, and Greptile were not run.

## Risks

Low risk. This changes a test wait only. It does not change delivery
behavior or database data. The existing global idle counter is valid for
this sequential fixture. If the suite later runs fixtures concurrently,
the wait may also wait for unrelated tracked work.

## Model Used

OpenAI Codex assisted with diagnosis, implementation, and validation.
The session configuration was `gpt-6.1-sol` with `xhigh` reasoning.
Read-only investigation and review used `gpt-6-luna` with `max`
reasoning. Browser preparation used `gpt-6.1-sol` with `medium`
reasoning. Tools included shell execution, Git, GitHub, Vitest, and
browser automation. The context window size was not exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [ ] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
 2026-10-29 15:58:27 -05:00
Nicky LeachandPaperclip 35ab90ad0f Map runtime activity to authorized task plugin panels (#18131) 
## Thinking Path

> - Paperclip manages agents and tasks.
> - Plugins contribute views to the task sidebar.
> - A ready view can belong to an earlier task.
> - Runner records activity under its authenticated run.
> - This PR maps that activity to installed task-panel declarations.
> - The sidebar checks readiness after current-run use and restores use
after reconnect.

## Linked Issues or Issue Description

Builds on merged contract #18129 and Runner publisher #18050. This PR
targets
master and is ready for review. Earlier combined verification is
retained in
[the verification
archive](https://github.com/paperclipai/paperclip/pull/18050#issuecomment-6489369676).

**Subsystem affected**

Run-event reads, plugin task contributions and the task sidebar.

**Problem or motivation**

Readiness polling can show a view left ready by another task. Reload and
reconnect must recover current-run use without losing concurrent live
activity.

**Proposed solution**

Map recorded activity to installed task-panel declarations. Merge
authorized
history and live activity, then perform bounded readiness checks for the
running agent. Preserve the existing read-only view.

## What Changed

- Add optional `{ namespace, activityKey }` activation mappings to
task-panel
declarations. Legacy declarations remain valid but contribute no task
panel.
- Add `view=plugin-panels` to the existing run-events route. Preserve
run-read and
telemetry-read authorization, secret redaction and private-run denial.
Map
activity to installed contributions and return at most 128 panel
identities.
- Scope activation caches to current panel versions and activity
declarations.
Preserve unchanged panels during unrelated plugin updates and history
reads.
- Merge history and live hints without losing concurrent activations.
Filter by
  company, task, run and agent. Restore history on mount and reconnect.
- Replace periodic availability checks with up to three readiness
attempts after
activity. Fresh activity restarts settled transient failures. Pending
checks,
  successful readiness and HTTP 4xx failures are left alone.
- Reuse live run updates with the panel cache key. Recover only panel
history
after a pending subscription read. Mark expected not-ready query results
so
  the global error reporter skips them; real failures remain reportable.
- Preserve current-run dismissal and the read-only plugin view. Update
manifest
  comments and producer/consumer documentation.

## Verification

- Current head: `5e04d757c`, based on master `09c05c550`. Range
comparison
confirmed all six feature commits were unchanged by the rebase. The
final
follow-up keys readiness by the plugin installation ID and availability
path.
- `cd ui && node ../node_modules/vitest/vitest.mjs run
src/hooks/useTaskPluginPanels.test.tsx`
— all 45 cases passed. `cd ui && node ../node_modules/typescript/bin/tsc
-b` passed.
- The shared query-client and live-update recovery selection passed 96
cases on
  `49c544cd2`. Token gates passed before the final hook-only changes.
- Earlier focused verification passed 57 manifest cases, 69
database/history cases,
event-registry cases and route authorization cases, including
private-run denial.
- Regression coverage includes subscription/history races, task changes
during
recovery, unfocused run completion, bounded readiness retries, denied
reads,
expected not-ready responses, changed activity mappings, unchanged panel
preservation, old cache scopes, and same-version route or installation
changes.
- [Current-head
CI](https://github.com/paperclipai/paperclip/actions/runs/40942001707)
passed: 54 successful checks, two skips, no failures or pending checks.
- [Standard and final
review](https://github.com/paperclipai/paperclip/pull/18131#issuecomment-6489842168)
both scored 5/5 on `5e04d757c`. The final review used the Apex tier,
added no
comments, and left no unresolved findings. The PR is conflict-free and
ready for review.
- Local checks stay targeted. GitHub CI provides the broad build, test,
typecheck,
browser and Runner coverage. No deployment or live-provider acceptance
is claimed.

<details>
<summary>Retained failures and recovery evidence</summary>

- `ac9329548` twice failed a browser reload assertion while conversation
history
was concealed. Master fixed its readiness wait in #17923. The approve
journey
then passed locally on `fea63a418` with installed Chrome (1 passed, 39
seconds).
All eight browser shards passed in later CI. The browser test is outside
this diff.
- A canary checkout failed with Git HTTP 408; its unchanged-head retry
passed.
- `de07d09fa` failed the mobile Cases card click after a ten-second
stability wait.
The same read-only journey passed locally on that exact head (1 passed,
26 seconds).
Its unchanged-head CI retry passed 22 cases, with one skip. The original
artifact
  is retained; the cause of that click timeout is not established.
- [CI on
`49c544cd2`](https://github.com/paperclipai/paperclip/actions/runs/40903679075)
passed 54 checks with two skips. CI also passed on `de07d09fa` after the
recorded retry.
- The `1e59ab441` merge checkout passed all browser and server suites.
Runner
shard 2 passed 2,252 cases and timed out waiting for a
controller-recovery command.
The test had landed on master in #18161 after the former branch base.
Rebasing
aligned the local checkout with that code; Runner shard 2 then passed on
`fcf7380d3`.
The original timeout cause remains unproven; no unrelated timeout was
changed.
- [CI on
`fcf7380d3`](https://github.com/paperclipai/paperclip/actions/runs/40940235193)
recorded runner shutdown signals in typecheck and a workspace shard. A
chat
recovery test observed a null wake deadline, and one Runner transport
fixture
failed during close. Those separate assertion failures remain
unexplained.
All of those jobs passed on the final readiness-key head. Those later
passes
establish recovery; they do not prove the cause of the earlier assertion
failures.
- Local browser setup first stopped at pnpm version-download
verification, then a
sandbox IPC restriction, then a missing bundled browser. Installed pnpm
9.15.4
  and Chrome completed the journeys. No dependency files were changed.
- Earlier optional-property TypeScript errors were corrected. Embedded
PostgreSQL
needed its install hook run from its package directory. Older CI also
recorded
fixture-cleanup lock timeouts, attention-fixture timeouts, and canceled
server
shards after runner shutdown. The parent fixed its FIFO snapshot-copy
failure
before merge. The linked verification archive retains earlier combined
evidence.

</details>

## Risks

Activity is a hint and grants no access. Missing plugins, denied reads
and
unavailable views contribute no panel. Older runs without activity
records show
no task plugin tab. A used view stays available for the running task;
reconnect
reconciles readiness. Matching producer and consumer builds are
required. No
migration, dependency, image publication or deployment is included.

## Model Used

OpenAI Codex, based on GPT-6, with reasoning, repository tools and code
execution.
The exact serving snapshot is not exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-29 13:56:48 -07:00
DottaandPaperclip 9e09d8f51f fix(runner): clarify native approval argument recovery (#18205) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - First-task onboarding requires a saved human proposal decision
before child work can start.
> - Native proposal tools reject interaction metadata placed outside
their advertised schema.
> - The existing rejection text focuses on question forms and does not
explain the misplaced approval metadata.
> - This pull request explains the field placement and keeps approval
recovery in its original confirmation or checkbox kind.
> - It also retains safe schema guidance in the run log so operators can
identify the rejected field.

## Linked Issues or Issue Description

Refs: #18025 for native first-task proposal recipes and #16889 for safe
native schema guidance. Related open PR #18105 changes separate server
API-client guidance.

**What happened?**

A native first-task confirmation copied `version` and
`supersedeOnUserComment` to the tool root. Both belong inside `payload`.
The runner correctly rejected the call before execution, but its
feedback described question forms. Later calls kept the misplaced field
and changed the approval into a question. No proposal card was created.
The run log retained only a generic schema error.

**Expected behavior**

Keep strict validation. Explain the nested field locations. Keep a
confirmation or checkbox approval in that kind while correcting its
arguments. Save the normal pending card and wait for an actual human
decision. Retain bounded schema guidance without storing rejected input.

**Steps to reproduce**

1. Send an otherwise valid native confirmation with root-level `version:
1` and `supersedeOnUserComment: false`.
2. Observe the pre-dispatch `invalid_tool_call` rejection.
3. Move both fields into `payload` and keep `interactionKind:
"confirmation"`.
4. Verify that the corrected call creates one pending confirmation,
exact replay creates no duplicate, and child creation still needs saved
human acceptance.

**Paperclip version or commit**

The observed argument shape and initial regressions came from public
source base `c32e1ba6`. The repair is based on maintained master
`4a6f6a65`. Public tests use synthetic prompts and identifiers.

## What Changed

- Give rejected confirmations and checkboxes approval-specific guidance.
It names `payload.version` and `payload.supersedeOnUserComment` without
suggesting a question-form recipe. Other human-input errors retain
complete question-form guidance and stable question/option ID
instructions.
- State the allowed native root fields in the first-task skill and put
the metadata inside both executable proposal recipes.
- Retain the existing safe guidance in `semantic_tool_denied`
diagnostics. Keep the original rejection identity and
`paperclipExecuted: false`.
- Preserve only the exact bounded scalar diagnostic payload after
verified event admission. The rest of the envelope and all API
request/result bodies keep their existing redaction.
- Add a real native Codex transport regression and strengthen
advertised-schema, card replay, human-admission, persistence, and
privacy controls. Update the execution contract and onboarding maps.

## Verification

- Red: the native transport reproduced the malformed confirmation and
showed that the old hint omitted the metadata correction. A second red
confirmed that approval guidance still included the question-form
recipe. The corrected transport case now rejects before execution,
retains a safe hint, and dispatches one corrected confirmation that
returns a pending card.
- Red: both shipped recipe checks lacked explicit nested metadata. All
49 first-task admission tests now pass, including advertised-schema
validation, idempotent card replay, actual human acceptance, agent-only
rejection, and question-answer controls.
- Updated native checks pass on the current repair: 95 native Codex
transport tests with 2 existing ignored cases, all 32 provider-bridge
tests, and the schema-feedback unit control.
- API/case persistence regressions were red before the event-boundary
correction. All 71 affected port/redaction tests pass in a clean run at
frozen head `3be3a70f`, including the corrected envelope privacy
regression. The focused 16-test persistence/privacy group and server
typecheck also passed earlier.
- Full local `pnpm build` and `pnpm -r typecheck` pass at exact head
`3be3a70f` in a clean detached checkout with Node 24.22.0 and pnpm
9.15.4. Exact-head CI passes: [full PR
CI](https://github.com/paperclipai/paperclip/actions/runs/40950360142)
and [isolated native Runner
readiness](https://github.com/paperclipai/paperclip/actions/runs/40950359835).
- The full local unit attempt recorded a 15-second timeout in the
unrelated Microsoft Teams 100,000-character existing-comment publication
case. That failure remains recorded. The exact isolated rerun passed (1
test; 1,814 unselected). The aggregate timeout cause is unproven. That
long local run straddled subsequent source edits and also observed the
two port regressions during their red phase; it was stopped and is not
final-head evidence. The clean affected-suite run above and current-head
CI provide the final verification. No timeout or assertion was weakened.
- Release-registry CI initially rejected the native test's compile-time
schema include outside the release crate's hashed input closure. The
guard was reproduced locally. The test now reads the current canonical
schema at execution. All 10 crate-cache controls and the real native
approval regression pass afterward; the cache guard and workflow keys
remain unchanged.
- Current-head Greptile review is 5/5 with no unresolved review threads.
All required checks pass (56 success, 4 skipped). The advisory Paperclip
Review remains neutral/incomplete after one normal re-review request; no
completed assessment or score is claimed.
- `cargo fmt --check`, map structural checks, and `git diff --check`
pass.
- Source and scripted-provider checks do not prove that a model will
follow the guidance. The original live onboarding attempt remains
failed; acceptance at a deployed revision is separate work.

## Risks

- This changes repair guidance and shipped instructions. It does not
make malformed arguments valid or supply a human decision.
- Diagnostic retention adds no rejected arguments, values, or unknown
input keys. The API receipt allowlist is unchanged. The validated-event
exception restores only the safe scalar payload; optional debug and
other envelope metadata still pass through redaction.
- The existing 512-byte guidance bound, strict tool schema, company/run
binding, idempotency checks, and proposal admission remain effective.

## Model Used

OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) through Codex, with tool use and code
execution. The context-window size is not exposed to this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-29 15:48:37 -05:00
Devin FoleyandPaperclip 5d79d88d68 fix(server): let settled native finalizations permit idle sleep (#18184) 
## Thinking Path

> - Paperclip manages AI agents and their work.
> - Native runs retain finalization records for recovery and audit
history.
> - Idle sleep checks durable work before it stops an instance.
> - The current check treats every retained finalization as pending
work.
> - A committed record can remain after its run and status effects have
finished.
> - This pull request recognizes complete records without deleting them.
> - Instances can sleep while unfinished execution and recovery work
remain protected.

## Linked Issues or Issue Description

**What happened?**

A retained `native_run_finalizations` row prevents idle sleep even after
the run has finished and its status decision has been applied. The
current gate checks whether any row exists.

**Expected behavior**

Complete finalization history should remain available without counting
as pending work. Active leases, retries, accounting, recovery, and
incomplete status effects must still prevent sleep.

**Steps to reproduce**

1. Finish a native run and retain its committed finalization.
2. Confirm that its accepted result, assessment, applied decision, and
delivered effects agree.
3. Request an idle report under a valid quiet admission hold.
4. Observe that the retained row reports background work before this
change.

**Paperclip version or commit**

Reproduced on master at `cd0c86e5e1d427c8fa8aa8c3a9499391ee9b425b` with
a local PostgreSQL fixture.

**Deployment mode**

Server with PostgreSQL. No provider calls are needed to reproduce the
gate failure.

Related PR and issue searches found no duplicate for this check.

## What Changed

- Replace the blanket finalization-row check with a completion proof
bound to the same company, issue, run, accepted result, assessment, and
applied decision.
- Require the owning run's successful workspace-finalization receipt;
retain unresolved leases, retries, accounting, recovery, and status
effects.
- Share final-reply candidate discovery with the presentation recovery
worker, so a completed run cannot hide a required reply. The pure query
stays with native presentation; idle inspection uses it in the existing
read-only transaction and five-second statement limit.
- Register open warm sessions and in-flight close/checkpoint promises
with local idle inspection. Historical controller/provider process
metadata remains preserved.
- Add PostgreSQL and local-warm regressions and document the
retained-history rule.

## Verification

- PostgreSQL sleep/finalization and chat-monitor continuation suites:
300 tests passed, including 56 finalization cases.
- Local-warm regression: passed; an open session and its pending
close/checkpoint both block sleep until retirement completes.
- Server typecheck and server build: passed after review fixes.
- Full workspace typecheck and build: passed before the review
additions; hosted checks cover the updated revision.
- An earlier local full invocation encountered two chat recovery
failures that passed in isolation, and two existing Git workspace
fixture failures caused by Xcode system Git initializing `main`. Those
two pass with `init.defaultBranch=master` scoped to the test subprocess.
That superseded full invocation was stopped for review revisions; no
full local pass is claimed. No machine configuration or unrelated source
was changed.
- Fresh Apex review: 5/5 on `6846203fe2383bae1ede888fbaaedf83e9946e73`,
with all findings addressed.
- Hosted CI: fully green on `6846203fe2383bae1ede888fbaaedf83e9946e73`,
including the final verify gate. Twelve jobs were interrupted by a
simultaneous runner shutdown; rerunning the failed jobs completed
successfully.
- No hosted deployment of this change is claimed.

## Risks

This changes an automatic-stop gate. An incomplete proof could stop an
instance with pending work. The SQL fails closed and retains the
existing admission, local work, plugin, schedule, backup, and
provider-allocation gates. PostgreSQL cases cover unresolved leases,
retry/recovery intent, accounting, projections, rejected results,
unapplied decisions, and pending effects.

No migration or history deletion is required. Reverting the predicate
restores the conservative gate.

## Model Used

OpenAI GPT-6 through Codex, with repository tools and local PostgreSQL
verification.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-29 11:11:22 -07:00
DottaandPaperclip 9f3fc2094f Fix shared workspace seed capacity and trace deferred copyback (#18181) 
Apply the existing company quota to company-shared workspace seed buckets. Preserve individual seed limits and recheck size after verification.

Trace each deferred native workspace finalization check and its resumed copyback under the same hashed run ID. Keep existing admission, ownership and retry behavior.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-29 12:44:34 -05:00
DottaandPaperclip f1911a4c53 perf(skills): move repository packages out of Postgres (#18082) 
<!-- entire-trail-link-start -->
https://entire.com/gh/paperclipai/-/trails/205/changes/paperclip/205
<!-- entire-trail-link-end -->

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agents receive selected company skills before each turn.
> - Imported skills can need files from the whole source repository.
> - The current path stores those files in Postgres and reads large JSON
payloads during preparation.
> - This pull request stores verified packages in the configured file or
object store and keeps preparation queries small.
> - It also converts old rows automatically and restores missing files
from the exact saved commit.
> - The result reduces database traffic while preserving skill versions
and repository dependencies.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Company skill import, version storage, and preparation before agent
turns.

**Current behavior**

Repository snapshot bodies live in JSONB. Skill preparation can read
those bodies again even when the files already exist on disk. Historical
packages and native assets also need a bounded cleanup path.

**Proposed behavior**

Store package archives and manifests through the existing storage
service. Keep version identities, receipts, hashes, and recovery
metadata in Postgres. Convert legacy rows after the stored package
passes read-back verification. A missing package can use legacy bytes or
its exact Git commit under current repository authorization. No separate
data-migration command is required.

**Reason and benefit**

The warm path sends no repository bodies through SQL and reads no object
bytes. Cleanup runs through the existing deadline coordinator. Current
versions, active runs, explicit pins, and authored content retain their
protection.

**Breaking changes**

The normal application schema upgrade is required. Single-version and
skill-file responses keep their existing shape. Version lists,
current-version summaries, and test-run version summaries now contain
file names and kinds without file bodies. Diff and restore fetch
selected versions through the existing single-version endpoint. Older
server builds must not write after conversion unless they support the
new package format. The storage override and rollback limits are
documented in `doc/skill-package-storage.md`.

**Additional context**

Related work: #15731 preserves repository dependencies; #17923 changes
version-cache scheduling; #18080 changes delivery to remote Codex homes.
This change owns package persistence, acquisition, and expiry. It does
not replace those delivery or scheduler changes.

## What Changed

- Add company-scoped archive receipts, exact-commit recovery recipes,
migration deadlines, and a durable object-deletion queue.
- Add bounded archive upload, read-back verification, extraction,
concurrent request coalescing, and verified local caches.
- Convert old snapshots and versions automatically. Keep legacy bytes
when verification fails.
- Recover missing files through current source authorization and the
original commit. Validate repository identity, file hashes, modes, and
the aggregate hash.
- Expire unused recoverable Git packages and native assets after 30
days. Preserve active pins and the durable copy of authored or unproven
content.
- Keep warm preparation queries metadata-only. Bound file digest caches
and avoid repeated body hashing when file identities are unchanged.
- Extend the existing storage contract with cancellation. Register
package work with the existing coordinator. Optional conversion retries
retain the original bytes and may wait for ingress without blocking
sleep.
- Add regression tests, operating guidance, a deep trace, benchmark
receipts, and an optional maintenance command.

## Verification

### Final master integration and review repair — October 29, 2026

Current head: `a6065424249e763dfeac37161f1b52a9f25bd9b7`. Commit
`f8bbdd3dbd` resolves the conflicts with master `76a796e964`; the final
two commits repair the Apex deletion findings below.

- Keep both package-maintenance and version-retention notifications in
their write transactions.
- Keep the shared run-pin predicate and master's controller-lease
deadline metadata. Package expiry reads the updated protected-version
result.
- Preserve every master migration entry. Generate migration **0415
(`0415_zippy_zemo.sql`)** after master's 0414. Its skill SQL is
byte-for-byte unchanged from the previously tested idempotent migration.
Retain only the five newest schema snapshots. Migration numbering and
safety checks pass.
- Repair failed file/folder deletion: preserve a complete temporary
subtree until transaction commit, restore on upload or SQL failure, and
restore callback failures before releasing the mutation lock. Both
PostgreSQL regressions reproduced missing files before the fix and now
pass. They cover binary data, executable permissions, empty folders,
unlisted files, unchanged database/version rows, successful retry after
upload-claim expiry, and historical reads.
- Reconcile lost commit replies under the same advisory and row locks.
Check the saved version and source identity before restoring. Keep
committed deletions intact and preserve backups when the database
outcome is unknown or a newer edit exists. Four real-PostgreSQL outcome
tests pass; the prior code failed three of them.
- Full workspace typecheck and production build pass on the final
commit. All **204** focused PostgreSQL package/cache/retention/authoring
and filesystem-boundary tests pass, with none skipped.
- Apex review **18 passes 5/5**, with zero new comments on the exact
final head. All **34** review conversations are resolved.
- [Final-head
CI](https://github.com/paperclipai/paperclip/actions/runs/40914116508)
completes successfully on attempt 2. The head has **55 successful
checks**, two neutral reports, two skipped checks, and a successful Snyk
status. No checks are pending or failing. All eight browser shards pass.
- Runner Vitest 2/2 timed out once in the unchanged ACPX
controller-replacement fixture. The same job passed on the preceding
commit. One unchanged CI retry passes all five ACPX recovery cases,
**2,253** Runner package cases and **126** native server integration
cases; their standard one and five skips remain. The initial timeout's
root cause is not established. Local diagnosis also hit the fixture's
eight-second session.open timeout with an authenticated Runner; no local
Runner pass is claimed.
- Both full local `pnpm test:run` attempts were stopped when Apex
identified deletion defects and source repair became necessary; neither
is claimed as a passing full local run. The final **204** focused tests,
full workspace typecheck, and production build pass. Full-suite
acceptance uses the final-head CI jobs. The superseded merge-head CI
included runners terminated by shutdown signals; each source repair
started a new full run.

- Auto-merge is enabled for squash merge. The final head is
conflict-free and all checks pass. GitHub's active master ruleset still
requires a code-owner approval for the dependency files. The existing
review requests are with `devinfoley`, `nickyleach`, and `forgottendev`;
no policy bypass was used.
- The earlier hosted stress and full native receipts below remain tied
to `a2e597d800`; they are not relabeled as a deployment of this final
integration. The earlier local PostgreSQL startup failure remains
recorded below; PostgreSQL ran successfully for this integration
verification.

### Full native skill execution — October 29, 2026

The full native-agent acceptance gap is closed on candidate
`a2e597d800eb989d565780e7aa947d5368676444`. Two qualifying native turns
used OpenAI API / `gpt-5.4-mini` in real Daytona sandboxes, on the same
disposable hosted TEST tenant as the stress test. A follow-up on the
same task consumed a newly saved skill version.

The task text asked for the assigned skill workflow. It did not contain
the proof nonce or expected receipt. The assigned authored skill
contained a Python verifier and a random package-local nonce. The agent
read the actual installed skill files and executed that verifier against
the imported, automatically converted 1,360-file repository package. It
checked the repository file count and reference/binary bytes, and
directly executed the repository's executable probe. Independent
verification matched the emitted JSON to the nonce and expected byte
hashes. The agent persisted the same receipt as its task response;
native finalization committed, workspace finalization succeeded, and
task state became Done.

| Qualifying turn | End-to-end run time | Recorded skills.prepare |
Repository files | Binary hex | Shell receipt SHA-256 |
|---|---:|---:|---:|---|---|
| Initial task | 47.881 s | 1822 ms | 1360 | `00ff07800a` |
`a1deb30dc095f2306c7aa9cc5ec396da12c68961cf5b809f4d7c20f7fc68493b` |
| Corrected follow-up, new nonce/version | 37.974 s | 377 ms | 1360 |
`00ff07800a` |
`dbcdc33fa1376b4edc136feae1cdf770560a73292467e93619efdda20e4282a7` |

Both successful shell receipts reported `probe: STRESS18082` and
`assetSha256:
e8ea7a2f15f7bb3b332a46adbef848465884c8996595aa234748befd544d1f3a`. Their
proof nonces, selected version IDs and receipt hashes differed, and both
ran on the same task. These are observed timings, not latency SLOs; they
include real provider work. The service stress-test timings remain a
separate measurement boundary.

Earlier failures remain in the evidence:

- The original five-minute Daytona allocation timeout never reached
model execution. After normal owned-stack restore, its durable cleanup
record was `expired` / `success`. The cause of that original allocation
timeout remains unknown; a later success establishes recovery only.
- Initial continuation setup used unsupported generic agent-update
fields and returned HTTP 409 before dispatch. Setup was corrected to use
the supported skill-assignment and versioned instruction commands.
- The intervening completed native turn read repository bytes and ran
the probe, but omitted the verifier JSON under the older test agent's
overly-specific instruction file. Its strict receipt validator failed;
it is not counted as a qualifying turn. Correcting the test agent's
instruction file and sending an explicit task-scoped wake produced the
qualifying follow-up above.
- The pinned application's immutable rate card omits `gpt-5.4-mini`, so
completed usage triggered the normal block-on-unpriced budget policy.
Supported append-only accounting corrections estimated the three
completed runs using the [official OpenAI model rate
card](https://developers.openai.com/api/docs/models/gpt-5.4-mini): $0.75
input, $0.075 cached input, $4.50 output per million tokens, assuming
standard processing. Estimates totaled 12.482655 cents. Original usage
and missing-price records are preserved; the $1 hard stop and
unpriced-usage guard stayed active.
- Model shell attempts included a missing `xxd` command and incorrect
guessed file paths. Those failed calls and subsequent successful actual
verifier calls remain visible in complete event streams. No
zero-tool-error claim is made.

No product source changed during this continuation. The previously
measured contention, cold-start and memory limits remain. The later
merge resolution is covered by the separate integration checks above;
these hosted receipts qualify their recorded candidate.


### Hosted stress test — October 29, 2026

**Hosted acceptance:** full native-agent execution and a changed-skill
follow-up pass on the recorded hosted candidate. The final integration
section above records the subsequent conflict resolution and its
separate checks.

Tested baseline `929e840595c8933ca485ac7a2e99413e1e11ca64` and this
exact candidate sequentially on one disposable TEST tenant, with
external Neon PostgreSQL and actual S3. The fixture contained **1,360
files / 22.4 MB / ten skills sharing one repository**. Measurements
cover deployed skill-preparation and native-context services in an
adjacent diagnostic process, not a complete agent turn. The container
had **24 vCPU / 24 GB**.

| Warm workload | Median before → after | p95 before → after | DB
returned value bytes/preparation before → after |
|---|---:|---:|---:|
| One skill, serial | 1,060 → 125 ms | 1,303 → 155 ms | 22,610,389 →
1,771 |
| One skill, four concurrent | 4,415 → 297 ms | 5,143 → 328 ms |
22,610,389 → 1,771 |
| Ten skills, serial | 5,723 → 535 ms | 6,057 → 600 ms | 224,739,030 →
10,671 |
| Ten skills, four concurrent | 24,410 → 1,986 ms | 24,833 → 2,221 ms |
224,739,030 → 10,671 |

- Candidate warm preparation performed **zero S3 GETs**. Ordinary-pool
queries fell from 16 to 9 for one skill and 61 to 9 for ten, excluding
dedicated lock connections. DB numbers measure returned values before
Drizzle decoding, not wire bytes. Baseline/candidate sample counts were
30/100, 32/100, 5/50 and 8/48 respectively; the small baseline ten-skill
samples are comparative evidence, not SLO estimates.
- **Application-UID soak:** 800 ten-skill preparations over 10.3
minutes, four workers paced by 2.5 seconds; **zero failures**, median
**559 ms**, p95 **775 ms**, p99 **910 ms**, zero warm S3 reads. All 726
accompanying health/company-list requests returned HTTP 200. Their
end-to-end p95 was 569/502 ms, including network latency. The probe
overlaps most of the soak. Soak DB-response capture was limited to its
first 60 seconds.
- **Important remaining costs:** an unpaced 16-way ten-skill burst had
p95 **8.32 seconds** because company-level preparation serializes. Cold
preparation took approximately **1.1–2.5 seconds** for one skill and
**7.4–10.0 seconds** for ten. Ten-skill diagnostic RSS remained about
**1.17 GiB** (baseline 1.18 GiB), and the soak peaked at **1.35 GiB**. A
concurrent whole-container sample was 3.45 GB. This does not establish
suitability for a small-memory host or eliminate per-entrypoint native
asset duplication.
- **Upgrade without a manual data migration passed:** legacy snapshot
JSON shrank from 22,465,691 bytes to `[]`, with a stored package and
zero pending conversions.
- **Recovery passed:** sixteen concurrent cold requests coalesced into
two S3 GETs; full bytes/binary/executable modes matched; same-size
corruption repaired; an injected storage failure preserved warm access
and failed cold access safely. A disabled public-Git source denied
recovery, and after the actual five-minute backoff its exact SHA
reconstructed all 229 files. Cross-company descriptor access was denied.
Private-Git credential revocation was not tested.
- **Retention passed:** reclaimed 27,754,496 allocated cache bytes,
deleted an unreferenced package row and both objects, preserved aged
authored history, and expired/reacquired reconstructible Git content. A
version pinned by the failed native run was correctly retained. Only
owned fixture timestamps were aged.
- **Browser UI passed:** authored skill creation, saved edit surviving
reload, version history, and restoration of earlier content.
- **Initial native attempt (superseded by the qualifying turns above):**
one real run failed at Daytona allocation after five minutes
(`setup_failed`), before any model execution. Provider cleanup also
reported failure. That attempt does not count as a passing native turn.
Its receipt and all evidence remain preserved; its cleanup later
recorded success after normal restoration.
- Initial A/B commands ran as SSH root. Diagnostic-owned lock/cache
permissions caused one setup HTTP 500 and the first UID-1000 soak to
fail; exact fixture paths were repaired. The final soak, retention and
recovery ran as application UID 1000. A fresh UID-1000 one-skill batch
also passed (30 samples, median 116 ms, p95 321 ms, 1,771 DB value
bytes/preparation). Earlier harness-export, SSH, npm provenance, backoff
and UI-wait failures remain recorded alongside their successful retries.
- Private evidence includes raw timings, assertions, HTTP probes,
screenshots and failure/cleanup receipts. Initial failed-campaign
cleanup suspended and retained the owned TEST tenant and released the
shared QA account lease at **15:22:41 UTC**. The tenant was subsequently
resumed under its existing ownership for the native execution
qualification above. Final cleanup completed at **15:53:09 UTC**: the
account lease is released and the tenant is suspended/retained under the
original October 30, 12:24:52 UTC retention deadline. Its separate stack
lease remains held; no active or pending-cleanup environment lease
remained. The original retention window is deliberately preserved even
after successful continuation.

- Earlier hosted-tested head:
`a2e597d800eb989d565780e7aa947d5368676444`, integrated with master
`929e840595`. The earlier merge conflicts were resolved at this head.
[Current-head
CI](https://github.com/paperclipai/paperclip/actions/runs/40884087629)
passed; all checks are complete with 55 successes, two neutral reports,
two skips, and no failures. [Apex review
15](https://github.com/paperclipai/paperclip/pull/18082#issuecomment-6480257258)
passed 5/5 with zero findings on this exact head. All review
conversations are resolved.
- Ten real-filesystem boundary cases and server typecheck pass locally
for the security fix. Coverage includes cache roots, descendants,
individual files, symlink aliases, recursive ancestors, a future
concurrent cache fill, and allowed sibling directories. [Current-head
private-project integration
tests](https://github.com/paperclipai/paperclip/actions/runs/40884087629/job/123483306405)
passed all 29 cases, including authorized and unauthorized imports of a
cached private package. Full typecheck, build, server suites, all eight
browser shards, and Runner checks passed in current-head CI. Local
database tests could not run because PostgreSQL could not initialize;
this limitation is recorded below.
- The earlier generated migration preserved the newest five snapshots.
The final integration renumbers its unchanged SQL to 0415. Migration
safety/numbering checks pass. Its SQL is byte-for-byte identical to the
previously verified idempotent skill migration; repeated application on
this latest schema was blocked by the same local database startup
failure.
- The broader verification before the final migration-only rebase passed
1,025 package, lifecycle, remote-launch, native-session, and
asset-retention tests, 52 focused UI tests, full workspace
typecheck/build, token gates, and map checks. The linked complete CI run
passed before the latest focused fixes.
- A disposable MinIO instance passed multipart upload, verified
read-back, and cold restoration of an 18 MB archive.
- Twenty warm samples with a synthetic 1,349-file, 22.8 MB repository
retained five metadata queries and zero object reads. Median preparation
was 256 ms for one selected skill and 691 ms for ten, with 5 ms emulated
network delay each way. These are local measurements, not production
latency guarantees.
- No production instance was changed. Earlier verification failures and
their disposition are retained below.

<details>
<summary>Earlier failures, review findings, fixes, and rerun
evidence</summary>

- The first CI pass found a stale inventory-span expectation and a
missing repository-read span. Both are fixed; all three observability
tests pass. One browser test copied page text into its YAML parser after
reload; its first unchanged-head retry passed, but the race recurred
after rebasing. The test now waits for the renamed URL and completed
reload before copying the editor, retaining its original assertions. The
earlier full local suite did not complete cleanly: it reported setup
timeouts and assertions, while isolated relevant reruns passed. The
complete current-head CI run closes that broad verification gap. No
production instance was changed.

- The first Apex review scored 2/5 and identified six issues. The fixes
protect authored history even when it shares an in-flight Git upload,
recheck upload expiry during orphan deletion, bound failed-upload
cleanup, retain per-version kind/encoding metadata, add sanitized
failure diagnostics, and correct the migration guide. The final focused
run passes all 140 tests; the preceding eight-file run passed all 149
tests, including local storage and S3 diagnostics. Server typecheck and
migration safety/numbering checks pass after the fixes. The second Apex
review scored 1/5 and found unbounded maintenance storage calls and
mismatched package/Git batch limits. Those are fixed with a 30-second
cancellable cleanup deadline, shared 100 MiB/10,000-file scanner and
storage limits, and bounded Git read batches. The proposed manual
lockfile edit was rejected with evidence of the enforced CI-owned
lockfile policy. All 19 storage/recovery unit tests and server
typechecking pass. The new database-backed cleanup/conversion tests were
locally skipped because the host exhausted its PostgreSQL semaphore
capacity (confirmed directly by initdb). That gap is closed by
current-head CI: [package-store
tests](https://github.com/paperclipai/paperclip/actions/runs/40771320775/job/123109300889)
pass all 18 cases and [package-worker
tests](https://github.com/paperclipai/paperclip/actions/runs/40771320775/job/123109301157)
pass all 7, with no skips in either suite. The corrected YAML browser
shard passes. A separate planning-fixture test failed its cleanup DELETE
with an activity-log foreign key violation after its behavior assertions
passed; its one unchanged-head retry passed. The third Apex review
scored 1/5 and identified a one-slot pool deadlock, projected path
limits, cleanup holding the preparation lock during archive reads, and a
publication/expiry race. These are fixed with transaction-local reads,
prefix-aware bounds, isolated verification, and shared package row locks
with a publication grace. The separate Paperclip review also found stale
historical inspection, eager history downloads, and conversion retries
blocking sleep. Those are fixed. Local PostgreSQL capacity recovered:
the new database regressions ran, including the one-slot fork, paused
storage during a warm turn, and a locked publication during expiry. The
expanded run passed 144/145 tests; its remaining assertion still
expected file bodies from the version list and was updated to fetch the
selected version. The corrected company-skill suite passes all 81 tests,
and the final package/projection run passes all 27. All 52 focused UI
tests pass after updating their query-provider fixtures; the first UI
run exposed a fixture remount, and the follow-up exposed an editing
syntax error, both corrected. Full workspace typecheck and build, token
gates, and map checks pass. The sandbox initially blocked the build
tool’s local IPC socket; the authorized build rerun passed. Master then
claimed migration 0401. The branch is rebased on f8226b0515, the
unchanged SQL is regenerated as 0402, and repeated application passed
twice on the combined schema. Post-rebase workspace typecheck passes.
All 1,025 package, lifecycle, remote-launch, native-session, and
asset-retention tests pass on the combined head. All CI checks passed
for head 9d80a3b52d. The fourth Apex review scored 3/5 and found a
save/edit lock-order cycle, a retry retaining an uncommitted source ID
in its recovery recipe, and a summary read racing conversion. Commit
b7d9237c6a fixes all three. The new concurrency and retry tests pass
within all 61 package/source tests, and server typecheck passes. Master
then claimed migration 0402. The branch is now rebased on 65fa1cd617 and
the unchanged SQL is regenerated as 0403. All 68 package, source, and
worker regressions, server typecheck, migration safety/numbering, and
repeated migration application pass on current head 62553671a0.
Subsequent verification and review outcomes are recorded below.


- The fifth Apex review scored 4/5 and found cached version reads
unnecessarily depending on object storage and conversion reads missing
the coordinator cancellation signal. Commit 1eb90d80be fixes both:
cached reads verify bytes without the preparation lock; manifest and
archive reads carry cancellation through conversion. The first focused
run passed 70/71 tests, including all three new cases, but the existing
diagnostic test observed no warning. Its fixture now sets an explicitly
expired migration deadline; the final worker suite passes all nine
cases. Server typecheck passes.


- The sixth Apex review scored 4/5 with one finding: concurrent cached
inventories were unbounded. Commit 4b7589e14c gives cached inventories
and previews a two-reader/32-waiter limit independent of cold storage.
The first expanded pressure test observed a third reader before the
preview path had the same bound; both paths now share the local limit.
All 73 focused cases pass, followed by all 29 package cases after the
final overload adjustment; server typecheck passes. The separate
Paperclip review found no must-fix or should-fix issues. Its optional
suggestion to persist durability verification was considered: cleanup
intentionally verifies current external bytes before discarding the last
local authored copy, and the missing-object regression requires that
check. Shared-package eviction happens on cold fills with at most two
attempts and a ten-second budget.


- The seventh Apex review reached 5/5 with one remaining P2:
timeout-bearing manifest reads bypassed request coalescing. The security
review also found repeated archive acquisition after a cold
administrative read. Commit 79a80a39d2 shares manifest loads with
per-caller cancellation and cancels the underlying work when the last
caller leaves; coalesces identical cold version/file requests; and
publishes verified restored packages into the shared cache. Publication
uses a short nonblocking lock, while eviction verification runs outside
it. The first expanded run passed 73/74 tests and exposed a missing
originating DB for cache publication inside read-only inspection;
transaction readers now pass that owner explicitly, keeping metadata on
the transaction. The corrected run passed 75/75. A final eviction test
initially expected the shared package to remain after only its separate
runtime projection was reused; it now explicitly accesses the shared
package during verification. All 32 final package tests pass, including
independent caller cancellation, one acquisition across
concurrent/repeated reads, and warm preparation during stalled eviction
verification. Server typecheck passes.


- The eighth Apex review scored 4/5 with one finding: decoded cold
inventories outlived the storage semaphore while waiting for cleanup.
Commit 9d7921d1ec adds a separate two-response/32-waiter lifetime bound
held through cleanup, preserving the independent warm-read queue and
forwarding cancellation through eviction verification. The new
regression pauses cleanup after two responses decode, verifies a third
archive stays queued, and completes warm preparation before releasing
cleanup. All 77 package/source/worker cases and server typecheck pass.
On the preceding head, serialized server shard 6/9 was interrupted with
“The operation was canceled” after 57 seconds of tests; the log contains
no failing assertion. Fresh latest-head CI reruns that shard along with
the other gates.

- The ninth Apex review scored 4/5 with one P1: cold responses could own
both permits while waiting on a one-slot ordinary database pool held by
a queued transaction reader. Commit a94f0e409f reads caller-scoped
metadata before admission and isolates all acquisition/cleanup SQL on a
private connection. The new real-PostgreSQL regression reproduces the
deadlock against the previous head and passes with the fix; all 78
focused cases and server typecheck pass. The previous CI run completed
with 54 successful checks, including both formerly interrupted/stuck
serialized shards.

- The tenth Apex review scored 4/5 with one P1: preloading legacy bodies
before admission could retain many full repositories in queued requests.
Commit 276e353a37 removes that preload and also bounds legacy manifest
derivation/fallback, using private connections inside the I/O slot. All
80 focused tests pass; the final synchronized package suite passes all
36 tests and server typecheck passes.

- Apex review 11 returned 5/5 with zero comments on 276e353a37. While it
ran, master 8bc051245f added migration 0403, creating a metadata
conflict. Merge commit e1783764d9 integrates master and generates
0404_clumsy_leo.sql; its skill SQL is byte-for-byte identical to the
previously verified migration. The combined schema passes migration
safety/numbering, repeated SQL application twice, all 80 focused
PostgreSQL tests, full workspace typecheck, and full build. The
preceding a94f0e409f CI run was superseded and cancelled while its final
runner job was active; no old-head success is claimed for that run.

- Final acceptance: Apex review 12 returned 5/5 with zero comments on
e1783764d9. All 28 review threads are resolved. The final head has 55
successful checks, two neutral reports, two skipped checks, and no
pending or failing checks; the PR workflow completed successfully.
Superseded run 40789121902 was cancelled to free its workflow slot for
the final head; normal cancellation did not finish promptly, so force
cancellation completed it.

- October 29 conflict resolution (867c00673b): master 929e840595 added
installed-selection reuse and bundled relocation, routine cleanup
notifications, a hardened YAML copy helper, and migrations through 0413.
The merge retains those behaviors together with package read scoping and
the public-selection fast path; restricted selections still reauthorize
and the full row read rechecks scope. Generated migration 0414 retains
the existing SQL unchanged. Workspace typecheck/build and migration
checks pass. Local database verification is unavailable: initdb reports
semget / “could not create semaphores: No space left on device”; all 168
cases skipped, and repeated migration application failed at startup.
Docker returned HTTP 500 during container creation, so no Docker test
pass is claimed. Current-head CI must close this gap.

- Apex review 14 scored 3/5 with one P1: a local import could copy
cached private package files into a company-wide skill. Commit
a2e597d800 rejects cache trees and ancestors through a canonical
filesystem boundary, including aliases and ancestors checked before a
concurrent fill. Ten local filesystem tests and server typecheck pass.
The existing private-project integration case now rejects package-cache
imports for both authorized and unauthorized users. The preceding merge
commit’s complete CI run passed, and logs confirm all 168 locally
skipped cases executed successfully: package store 36, sources 35,
package worker 9, runtime selection 4, observability 3, authoring 81.

- Apex review 15 returned 5/5 with zero new findings on a2e597d800. The
separate Paperclip review found no must-fix or should-fix issues. Its
optional suggestions were considered: unexpected filesystem cleanup
failures remain visible, the durable marker cannot replace checking that
bytes still exist before deleting a last local authored copy, and
administrative history batching remains outside the pre-turn
optimization. The cache-path literal centralization suggestion is also
optional; the current producer and guard use the same directory, covered
by the new integration regression. All review conversations are
resolved.

- Earlier October 29 CI/Apex acceptance: a2e597d800 was mergeable at
that check, Apex review 15 passed 5/5 with zero findings, and all review
conversations were resolved. The hosted verification above records the
later acceptance limits and renewed conflicts. Workflow 40884087629
completed successfully, with 55 successful checks, two neutral reports,
two skips, and no pending or failed checks. The project-skill-sources
job log confirms all 29 cases ran and passed, closing the latest
integration gap. The preceding merge workflow also confirmed all 168
focused database cases ran and passed. No local PostgreSQL or Docker
pass is claimed.

</details>

## Risks

- First use after eviction can require storage or Git access. Revoked
grants, removed sources, and unreachable commits can prevent
reconstruction. Recovery never substitutes a newer commit.
- Historical packages without a proven recipe keep durable bytes.
Cleanup favors retention when a run pin cannot be interpreted safely.
- New uploads and conversion use bounded work and persisted retries.
Large initial conversions still add temporary storage and database load.
- Optional cleanup can wait until the next server wake. Supported
notifications assume one owning server process.
- Per-entrypoint native tree duplication and cold preparation cost
remain. The final ten-skill cold sample took 47.6 seconds on the shared
host.
- Generic authored edits can still hold an existing edit transaction
while a package uploads.
- Explicit file/folder deletion uses temporary disk space for rollback.
An uncertain commit with a changed skill or unavailable database retains
that backup and logs its location for recovery.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, repository tools, code
execution, and tests. The exact served model ID and context window are
not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-29 12:19:37 -05:00
Nicky LeachandPaperclip 247de89eae fix(runner): preserve warm tasks during recovery (#18169) 
## Thinking Path

> - Paperclip manages agent work and the lifetime of its execution
resources.
> - A warm native task can keep one Runner alive between successful
turns.
> - Foreground completion selects whether that resource stays running or
stops.
> - Background recovery must use the same selected policy.
> - Recovery currently reads that policy only from workspace copy-back
records.
> - A task without such a record can be stopped immediately after
foreground retention.
> - This pull request saves the resource policy with the native runtime
and permits the retained task to refresh its control connection.

## Linked Issues or Issue Description

**What happened?**

Background native finalization can stop a successful warm Runner when
its run has no Core workspace-sync record. Foreground completion retains
that same task. The stopped Runner cannot receive the later drain and
suspend commands. Cold resume then correctly rejects its unproven
checkpoint.

Hosted diagnostics confirmed the recovery reconciler attempted
environment release immediately after foreground retention. The provider
connection returned HTTP 404 within the same second. A database-backed
regression reproduced the stop without a workspace-sync record.

A separate authorization gap also prevents the current retained task
from refreshing its connection endpoint after the run succeeds.

**Expected behavior**

Foreground completion and recovery must use the selected resource
policy. A successful warm task stays running until its idle shutdown can
suspend it. Failed or cancelled work still stops. The current retained
task can refresh its existing control route, but cannot prepare or
submit new execution.

**Steps to reproduce**

1. Select a warm native runtime on a reusable remote execution
environment.
2. Complete a turn without a Core workspace-sync record.
3. Let native finalization recovery settle the still-active run lease
before foreground retention finishes.
4. Before this change, recovery defaults to `stop_and_retain` and
invokes the provider stop path.
5. Separately, request the retained task's connection endpoint after
successful completion. Before this change, Core rejects that request.

**Paperclip version/commit**

The hosted failure used Core 31b927e301.
Regressions also reproduced on master at 6d59f5ef7.

**Deployment mode**

A remote native execution environment with reusable warm tasks. The
resource policy is generic and applies across harnesses and providers.

Related recovery coverage: Refs #18161. This change preserves the
suspended-checkpoint admission guard. Existing tasks without the newly
saved policy retain the previous conservative fallback; deployments must
drain or recover them explicitly.

## What Changed

- Save the selected native resource disposition in the existing
runtime-selection transaction. Add no startup database round trip or
network request.
- Read this small policy in background settlement and recovery when no
workspace-sync record exists. Preserve valid workspace-sync policies and
export-retry behavior.
- Keep failed and cancelled runs out of warm retention.
- Permit connection refresh for a successful, dispatched warm task on
its current retained reusable lease. Keep retirement, termination,
expiry, saved-run identity, and plugin capability checks.
- Add database-backed retention and selection regressions,
failure/cancellation coverage, and bounded projection checks. Document
the lifecycle and access contracts.

## Verification

- Before the source edits, the no-workspace-sync regression invoked the
driver with `stop_and_retain` instead of retaining the lease. Both
runtime-selection cases failed because the resource policy was absent.
- After the source edits, those regressions and both
run-state/runtime-selection files passed. The three-file run had 77
passes and two new fixture expectation failures. Those fixtures were
corrected. Final-head GitHub CI passed, including the server suites,
Runner checks, typecheck, build, and E2E gates. The local rerun was
deferred at the user's request.
- Commands: `node_modules/.bin/vitest run
server/src/__tests__/native-environment-settlement-work.test.ts
server/src/services/heartbeat/runtime-selection.test.ts
server/src/services/heartbeat/run-state.test.ts`.
- The connection authorization regression failed before its edit. All 34
tests in `server/src/__tests__/environment-task-runtime.test.ts` then
passed.
- `node scripts/check-maps.mjs` and `git diff --check` passed.
- No comprehensive local suite was run. Further local tests and VM
reproductions were stopped at the user's request to conserve battery and
mobile data. GitHub CI verifies the final head.
- Hosted verification of this patch requires deployment after merge.
Passing pre-patch retries did not establish repair of the original
failure.

- Standard Greptile and the subsequent requested Apex review both
returned **5/5** on `360bcd7e789d65950c27484af237faec02553666`, with no
inline findings.

## Risks

- This adds one server-owned field to the existing run profile JSON; no
migration is required. Invalid or absent policy keeps the previous
conservative fallback.
- Valid workspace-sync disposition remains authoritative. Pending export
and failed finalization still use stop-and-retain safety behavior.
- Retained connection access requires a successful run, matching saved
run identity, dispatched warm task, unexpired reusable lease, and no
retirement or termination. Superseded leases and new execution remain
denied.
- The cold-resume checkpoint proof is unchanged. This patch does not
make the already-stopped task's unsuspended checkpoint resumable.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, tool use, and code
execution. The host does not expose a more specific serving model ID or
context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run targeted local checks; final verification passed in
GitHub CI (local rerun deferred at the user's request)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-29 09:19:49 -07:00
DottaPaperclipCodex
 		07c6df260b fix(server): schedule archived runner evidence from saved deadlines (#17925) 
<!-- entire-trail-link-start -->
https://entire.com/gh/paperclipai/-/trails/85/changes/paperclip/85
<!-- entire-trail-link-end -->

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Retired native sessions leave continuity and quarantine evidence on
disk.
> - The old hourly loop mixes evidence retention with backup expiry,
which has different database eligibility rules.
> - Evidence already has saved creation timestamps and local publication
signals.
> - This pull request gives evidence its own registered worker and
actual next deadline.
> - The worker stops filesystem and SQL activity when its evidence
obligations settle.

## Linked Issues or Issue Description

Stack slice 65. Parent: #17923
(`codex/sleep-64-skill-version-retention`). Related foundation: #15625.

**What happened?**

At parent `7b94aa185c29673372646bbf2e07a2fbd16bc394`, an hourly
backup/evidence tick still inspects native state roots. Evidence's
private due time is checked by that recurring tick instead of the shared
scheduler. Empty evidence and future retention windows have no reason to
be rediscovered every hour.

**Expected behavior**

Reconstruct evidence once. Wake from the existing publication signal.
Register the earliest saved retention deadline. Keep busy, failed, or
incomplete evidence recoverable. Preserve backup eligibility as a
separate duty.

**Steps to reproduce**

Start retention with an empty evidence root or archives inside the
seven-day window. The old timer still visits the combined owner. The new
evidence worker disarms when empty, or registers the saved expiry with
no evidence filesystem or SQL calls between relevant changes and that
deadline.

## What Changed

- Add an evidence worker through the delivery coordinator and shared
scheduler. Track startup reconstruction without delaying HTTP readiness.
Unsubscribe local publication before delivery shutdown.
- Select `backups` for the existing backup timer and `evidence` for the
new worker. Existing callers that omit the selection retain combined
behavior. Backup expiration remains the next separate migration.
- Use hourly retries for concrete busy or failed evidence, with earlier
saved expiries and one-second continuations independent. Restore
candidate/removal/byte summaries. Schedule publication at the first
removable instant; preserve legacy manual-tick semantics.
- Expose the existing earliest evidence deadline. Preserve one-second
bounded continuation, concrete failure/busy retries, cleanup-source
archive exclusions, symlink protection, late activity checks, atomic
rename, and interrupted removal recovery.
- Preserve non-ENOENT inspection errors as recovery obligations. Retry a
failed session root in a new evidence pass instead of continuing past it
and forgetting it.
- Register the first removable instant after the inclusive retention
boundary. Keep the existing manual dry-run publication semantics.
- Add real filesystem/isolated PostgreSQL lifecycle tests and
disabled/held-startup wiring cases. Add no table or migration.

The archive and quarantine writers already call
`noteNativeRunnerEvidence`; this PR connects that existing signal to the
scheduler. Shared code owns scheduling, while evidence rules stay with
the retention service. See
[`doc/plans/2026-10-27-runner-evidence-retention-sleep-audit.md`](doc/plans/2026-10-27-runner-evidence-retention-sleep-audit.md).

## Verification

- Adopted actual prerequisite squash
`b06ea111a254c62e0a55199b010d82e0adb33bf0` from #17923 through normal
merges. Qualified runtime/package/UI/map trees remain unchanged. The
final nine-path diff includes one browser-spec readiness repair.
- Runtime qualification remains **652 passing TypeScript cases across 22
complete files**, plus **29 passing Rust cases** in native-backend,
runtime-notification and configured-environment targets. One actual
closed Pi snapshot launch is intentionally skipped because its external
distribution root is not provided. Accepted source is
`b80eee32f344ae0218d8b791c14f2ea174da09cc`; all runtime and earlier
sleep-test bytes are preserved.
- The complete outage-recovery browser file passes **all five journeys**
on a fresh isolated database after four additive rendered-history
readiness assertions. Every original assertion, outage control and
timeout remains byte-identical when those new lines are removed. The
intentionally unresolved first-session scenario keeps its pre-auth
interaction and delayed-owner coverage. The prior CI/local timeout
evidence and one rejected broader precondition remain in the audit.
- Adapter/Runner/database builds, server/Runner typechecking, Rust
formatting/workspace checks, server build, module boundaries, migration
order and generated TypeScript/Rust profiles pass. PostgreSQL 17
fixtures apply all 413 journal entries through index 414; source hashes
and temporary fixture imports restore exactly.
- **Inherited validation exception:** Copilot profile 81's separate
source-identity check fails on the merged Rust configured-environment
hash. Its inputs and attestations equal upstream and stay unchanged
here. Grok's identity check passes. No release attestation is
regenerated or new live-provider qualification claimed. Required
exact-head CI/security and Greptile 5/5 still must pass.
- [The evidence
audit](doc/plans/2026-10-27-runner-evidence-retention-sleep-audit.md)
retains historical failures, counts and limitations. No full local
repository-suite, live provider, external-alarm, measured suspension or
fleet-savings claim. Fresh final-head CI/security, exact-head Greptile
5/5 and actionable Ponytail findings remain separate merge gates.

## Risks

- Live publication is the existing server-owned executor in one process.
Independent filesystem writers need an explicit signal contract. Startup
reconstructs restart state.
- Busy sessions and concrete inspection/removal failures retain retries.
Unknown cleanup-source archives remain protected.
- The existing scan enumerates directory names before applying its
entry/removal budget. This PR preserves that behavior and does not claim
a bounded total directory-listing allocation.
- An in-process deadline cannot wake stopped compute. Cloud sleep-safety
and external-alarm coverage stay unchanged. Physical evidence
reclamation can wait for a supported host wake.
- Backup expiration still has an hourly database eligibility scan. This
slice removes only the archived-evidence duty from that loop and does
not claim full instance suspension.

## Model Used

OpenAI Codex (GPT-6-based coding agent), using repository inspection,
normal Git merges, composition proofs and complete
PostgreSQL/filesystem/startup suites. No subagents were used in this
reconciliation.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Codex <noreply@openai.com>
 2026-10-29 11:12:16 -05:00
DottaandPaperclip c32e1ba6da fix(onboarding): keep approved child task handoffs runnable (#18160) 
## Thinking Path

> - Paperclip manages work through tasks, agent runs, dependencies, and
human decisions.
> - The first-task workflow asks for approval before it creates an
execution task.
> - File-producing work must run on a child task before onboarding can
finish.
> - A child blocked by its onboarding source cannot start while the
source waits for that child.
> - This change rejects that dependency during task creation and later
edits, and gives the agent a correction.
> - The shipped recipe shows the dependency direction and requires the
actual approved inputs in the child instructions.

## Linked Issues or Issue Description

**What happened?**

After human approval, an onboarding run could create a child with the
onboarding source in `blockedByTaskIds`. Creation saved a blocked child
with no wake. The parent could then wait for that child. A handoff could
also refer to input from the parent conversation without including the
input in the child description.

**Expected behavior**

The approved child can execute before its waiting parent. Invalid
first-task dependency input fails before task creation and leaves the
same command key available for a corrected call. The child instructions
include the required approved input and constraints.

**Steps to reproduce**

1. Accept a proposal on a server-created first task.
2. Call native `create_task` with `status: "todo"` and the onboarding
source in `blockedByTaskIds`.
3. Observe a blocked child with no scheduled wake. Block the parent on
that child to expose the stalled handoff.

**Paperclip version or commit**

The regression reproduces on `f8484a9f8d1f68e9534fee4f29e00e716dc680f4`.
The repair is based on current master.

**Deployment mode**

Core native and REST task creation. The same server boundary serves
self-hosted and hosted instances.

Related: #18080 fixed remote delivery of the selected first-task skill.
This change handles the later child handoff boundary.

## What Changed

- Reject a task blocked by its authenticated first-task source during
creation and later dependency edits. Later agent edits also check the
locked target's server-owned onboarding parent, protecting a child
running under its own execution. Compare UUID identity in PostgreSQL so
alternate supported UUID spellings cannot bypass the check. The error
explains the correction.
- Add a native child/dependency recipe with self-contained approved
input guidance. Clarify the general `create_task` input descriptions and
update their generated contracts.
- Cover native and REST rejection, corrected retry, preserved valid
dependencies and replay, and an executable recipe with exact input bytes
read from the child's own context. Thread the authenticated run through
native dependency edits. Keep blocker corrections independent of a new
proposal decision.
- Update execution semantics and the existing onboarding map entries.

## Verification

- Red on `f8484a9f`: the new native regression received a created
blocked child instead of the expected rejection.
- Review regressions were red on the original PR head: uppercase,
unhyphenated and braced source UUIDs created blocked children; a later
child PATCH returned 200 and saved the reversed blocker.
- Review follow-up reproduced both native and REST child-own-run
bypasses on `75fa75a6`: the native dependency call applied the reverse
edge; the child's own PATCH returned 200. The regression uses the actual
authenticated child execution and an already checked-out task.
- All 95 tests across first-task admission, assets, and the full native
tool-authority suite pass. The shipped recipe is validated against the
native schemas and exercised through real task/dependency services with
disposable PostgreSQL. Rejected PATCH tests preserve existing
authorization and attempt-audit behavior and require no requested
task-field, relation or wake change. Compatibility controls retain
ordinary manual parent blockers, explicit board edits, other blockers
and empty-list corrections.
- Runner TypeScript build, generated contracts, capability inventory,
protocol manifest, and map checks passed.
- Server typecheck and all 95 focused tests pass with the child-run
follow-up. All latest checks pass on `f24509ae`: 55 successful checks,
one neutral review check, four skipped checks, and a successful legacy
security status. Both automated reviews are 5/5 on this exact commit.
- Prior-head CI failures are preserved: the Runner close fixture had
unconfirmed provider drain while suspension succeeded; its exact
fake-provider subprocess row passed locally with unchanged sources. The
browser outage test lost its title textarea before any outage write. A
bounded local control with a fully loaded composer passed every original
outage assertion. The prior failure is an editor-readiness race; its
exact asynchronous ordering remains unproved. No UI, browser-test, or
timeout change is included. Neither failure is counted as a pass or
silently discarded.
- The first `f24509ae` CI attempt had two interrupted jobs: Typecheck
and General server 9/12. Their logs record runner shutdown signals. The
server job also records lost runner communication. The same-head
failed-job rerun passed both jobs and the required aggregate. It reused
the other successful results. No source, test, or timeout change was
made for the rerun. The failed attempt remains preserved. Full
repository checks pass in CI; the local full repository typecheck, test,
and build commands were not repeated.
- No external model or provider calls are used by these checks.
Real-provider acceptance at this revision remains separate.

## Risks

- The new rejection uses the authenticated run and the server-owned
onboarding origin. Ordinary tasks, other blockers, empty-list
corrections, and already committed creation replay retain their existing
behavior. Blocker edits do not reapply the creation approval gate.
- Nonempty blocker edits perform two indexed identity reads; the task
lookup covers at most the run source and locked structural parent. Other
patches and empty-list corrections add no source lookup. The update
guard adds no parent row lock behind a child. Existing REST
authorization checkout and attempt auditing remain in place before the
domain mutation guard.
- The explicit input recipe guides the model. Schema and persistence
tests do not prove that a future model follows every instruction.
- The tool guidance changes the generated semantic catalog identity. The
maintained generated files are synchronized in this change.

## Model Used

OpenAI `gpt-6.1-sol` through Codex, with `ultra` reasoning effort,
repository tools, code generation, and test execution. The context
window is not exposed by the runtime.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used and stated the available runtime
detail
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have described the issue in-PR following the bug issue template
- [x] I have not referenced internal or instance-local Paperclip issues
or links
- [x] My branch name describes the change and contains no internal
ticket ID
- [x] I have run focused tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-29 10:57:31 -05:00
Nicky LeachandPaperclip 6d59f5ef7e fix: allow workspace changes after settled native bootstrap failures (#18166) 
## Thinking Path

> - Paperclip manages work assigned to AI agents.
> - Workspace admission must wait for file persistence and execution
cleanup.
> - A native startup error can record a failed workspace finalization
before provider work starts.
> - The controller and environment can settle while that diagnostic
remains.
> - The workspace guard then blocks every later binding change.
> - This pull request recognizes settled bootstrap failures with no
output obligation.
> - Real output, process ownership, and cleanup obligations continue to
block changes.

## Linked Issues or Issue Description

Related: #14234 handles recovery-hold regeneration for legacy failures.
This change addresses the separate native workspace-finalization guard.

**What happened?**

A workspace change stayed blocked after two native startup failures.
Both runs had explicit bootstrap receipts with `providerWorkStarted:
false`. Their controllers were terminal and unowned. Their leases had
completed cleanup. No provider events or workspace copyback descriptors
existed. Each run still had a failed `workspace_finalize` diagnostic.
Later successful turns did not remove that historical record, so
selecting a task directory failed during setup on every retry.

**Expected behavior**

A confirmed startup failure with no provider work must not create a
permanent file-persistence barrier. A real workspace export or cleanup
obligation must remain fenced.

**Steps to reproduce**

1. Fail native session startup before provider work, leaving a failed
finalization diagnostic.
2. Settle the failed controller and its environment cleanup.
3. Schedule a different task workspace.
4. Start the next admission. Before this fix, it fails with “Previous
task run still owns workspace recovery.”

**Paperclip version or commit**

Reproduced against the workspace guard on master at `27be85904`.

**Deployment mode**

Hosted application with external PostgreSQL and a plugin-managed agent
computer.

## What Changed

- Exempt a failed finalization diagnostic only for finished native
bootstrap failures with an explicit no-work receipt and an unowned
terminal coordinator without a result.
- Keep provider events, process identities, restore failures, sync
descriptors, copyback owners, unfinished operations, and unsettled
environment cleanup as blockers.
- Reuse the existing batched provider-evidence reader.
- Test both pending workspace selection and legacy binding updates. Keep
the original failed operation unchanged.
- Document the admission invariant.

## Verification

- The new reproduction failed on both workspace-change paths before the
fix.
- `pnpm exec vitest run
server/src/__tests__/task-workspace-binding.test.ts`: 92 tests passed
after the fix.
- A first isolated test attempt needed the existing plugin SDK build
prerequisites. Tests ran after `pnpm --filter @paperclipai/plugin-sdk
ensure-build-deps`.
- `pnpm -r typecheck` and `pnpm build` pass with pnpm 9.15.4 and Rust
1.97.1. The initial typecheck attempt lacked Cargo on PATH; it was
corrected to use the installed toolchain. The unsharded local `pnpm
test:run` did not finish during the validation window and was stopped
after all PR CI test shards passed. It did not provide a completed local
full-suite verdict.
- Remote recovery used an audited, scoped transaction to append no-op
reconciliation records for the two verified bootstrap failures. Original
failures and workspace data were retained. The normal UI Retry then
applied the pending selection and reached native provider execution.
- Greptile reviewed commit `807d22be9e530e272d53def365a0cdd37bd142ea`:
5/5, with no review threads. All PR CI checks passed. GitHub reports no
merge conflicts.

## Risks

The main risk is misclassifying an unfinished workspace as a
startup-only diagnostic. The exemption requires positive no-work
evidence and rejects conflicting provider or output state. Existing
ownership and cleanup gates run first. Provider evidence adds one
batched database read only when eligible failed bootstrap diagnostics
exist. There is no new background work, migration, dependency, or
provider replay. Reverting this change restores the former guard.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, code execution, and browser
tools. This session does not expose a more specific model build ID or
context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-29 08:34:52 -07:00
DottaPaperclipCodex
 		b06ea111a2 fix(server): schedule skill cache retention from changes and saved deadlines (#17923) 
<!-- entire-trail-link-start -->
https://entire.com/gh/paperclipai/-/trails/83/changes/paperclip/83
<!-- entire-trail-link-end -->

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The server caches saved skill versions on disk so runs can restore
their exact instructions.
> - The old retention owner scans company rows every hour, even after
all caches are protected or removed.
> - Cache publication, version changes, and pin release already have
local or transaction-owned boundaries.
> - Grace windows and terminal controller leases already have saved
timestamps.
> - This pull request uses those changes and deadlines to drive the
existing bounded prune service.
> - Idle retention stops SQL and directory reads while pending cleanup
remains recoverable.

## Linked Issues or Issue Description

Stack slice 64. Parent: #17876 (`codex/sleep-63-agent-timer-deadlines`).
Related: #15625 establishes transaction-owned work signals. #17907
independently keeps an active cache-lock transaction alive; this PR
preserves that lock and does not duplicate that repair.

**What happened?**

At parent `31c12f1d68ff4a51e06fc2be2abf2c2aee9f9d3a`, skill-version
retention has an hourly company scan, a startup timeout, and separate
continuation timers. Empty or permanently protected caches still cause
recurring SQL.

**Expected behavior**

Reconstruct once, inspect changed companies, and schedule saved
access/staging grace or terminal controller deadlines. Stop recurring
work when no timed or uncertain obligation remains. Keep
current/newest/agent/run protection, restorable-byte validation, cache
locking, final pin revalidation, atomic rename, and trash recovery.

**Steps to reproduce**

Start an enabled server with no cached versions, or only current/pinned
versions. After the initial scan, advance an hour. The old interval
selects companies again. The new worker has no deadline and performs no
retention SQL or directory scan in that state.

## What Changed

- Signal the missing-local-source reconciliation deletion in its
transaction. Limit fresh scans of a successfully observed company to one
per thirty seconds. Keep one-second page continuations and actual saved
deadlines independent. Empty or uncertain passes do not delay first
publication.
- Restart a company scan if drain interrupts inspection or post-rename
trash cleanup. Restore bounded candidate/byte logging with dry-run mode.
Three controlled cases failed before repair and the complete 34-case
retention suite passes afterward.
- Register retention through the delivery coordinator and shared
scheduler. Remove the hourly, startup, and continuation timers from
`index.ts`. Track startup work without delaying readiness. Stop delivery
work before closing retained iterators.
- Keep one active company cursor and bounded 50-entry passes. Preserve
observed deadlines across pages and bound company discovery to 51 rows
per page. Retain concrete company failures and incomplete pins for
retry.
- Report saved grace and terminal controller deadlines from the existing
cache owner. PostgreSQL remains the controller-lease authority.
Current/newest/agent pins do not arm unrelated lease checks.
- Wake on cache publication, rejected materialization, and access
markers. Signal version creation/deletion and agent configuration/hire
changes in their owning transactions. Extend existing instruction,
finalization/environment, and retained-session fanouts with company
hints.
- Route reused provider-task prior-owner retirement through
`writeEnvironmentCleanup`, so that supported pin-release write also
settles a notification.
- Add an optional scope callback to `registerDeadlineWork`. Existing
consumers omit it and keep their behavior. The feature owns persistence
predicates; startup only registers it. Add no table or migration.

Full producer audit, shared-contract explanation, lifecycle evidence,
and limitations:
[`doc/plans/2026-10-27-skill-version-retention-sleep-audit.md`](doc/plans/2026-10-27-skill-version-retention-sleep-audit.md).

## Verification

- Reconciled all existing sleep changes with actual master
`7e27bc5007c8f9cb8cdce06c24f9813e27f7120f` through normal merges. All 24
retention-owned and 95 unrelated upstream paths are preserved; package,
UI, E2E and map trees equal master.
- Combined-source qualification: **1,009 TypeScript cases across 42
complete files**, plus **28 Rust cases** in two targets. One intentional
Linux-inotify-only adapter case is skipped on this macOS host. Coverage
includes actual retention/cache/producer/runtime behavior, startup,
shared deadline consumers, changed OpenCode/observability and native
execution boundaries.
- After the subsequent task-index/task-read upstream merges, **85 cases
across six complete files** passed on final source
`d9277b1858a2230bdb726974c90a0c9bc5c5ebe6`: real retention/cache/work,
task reads/query behavior and concurrent migration repair. Counts
overlap earlier coverage. Fresh PostgreSQL 17 schemas apply all 413
journal entries through index 414.
- Database/adapter/Runner builds, production server and Runner
typechecking, module boundaries, target migration order and generated
TypeScript/Rust release-profile consistency pass. Selected source hashes
and temporary fixture imports restore exactly.
- Reproduced and repaired Greptile's own-company due-cleanup P2 with
identical old-worker-failing/new-worker-passing assertions. The
30-second ordinary scan floor remains; already-due cleanup receives its
owed scan and disarms afterward. Earlier 799-case and fresh-blank-plugin
acceptance remain in the audit.
- **Inherited validation exception:** the separate Copilot
source-identity check fails after upstream OpenCode reserved-environment
changes; all 72 identity inputs equal current master. No provider
release attestation is regenerated or new live qualification claimed
here. Required exact-head CI/security and Greptile 5/5 still must pass.
- Historical results, repair details and limitations remain in [the
retention
audit](doc/plans/2026-10-27-skill-version-retention-sleep-audit.md). No
full local repository-suite, live provider, production-load,
external-alarm, measured suspend/wake or fleet-savings claim.

- CI on the prior published head found four connection-review assertions
racing the concealed conversation after reload. Its fresh
Greptile/Ponytail reviews were both 5/5, but the failing browser gate
was not waived or retried unchanged. The fixture now foregrounds the
task tab and waits for visible history before retaining all existing
saved-result assertions. The full file passes four journeys locally; the
controlled-restart case is intentionally skipped by the default
configuration. Original local journeys also passed unchanged, so no
deterministic local red reproduction is claimed. See the audit for CI
failure artifacts and scope. Runtime and prior sleep-test sources remain
unchanged.

## Risks

- Supported live writers are the existing service/API owners in one
process. Independent SQL/FS writers need an ingress contract. Startup
reconstruction covers restarts.
- Notifications cause a bounded inspection; they do not authorize
deletion. Durable pin predicates and the cache lock remain
authoritative.
- A database clock behind application time can require another
one-second check of the concrete expired controller lease. Unknown pins,
lock contention, and failed inspection retain retries until resolved.
- The in-process scheduler cannot wake stopped compute. This PR
preserves Cloud sleep-safety gates and external-alarm coverage. It does
not prove deployed suspension or fleet savings. Cache reclamation can
wait until a supported wake when compute is stopped.
- Native-harness backup retention and the other remaining duties are
separate stack slices.

## Model Used

OpenAI Codex (GPT-6-based coding agent), using repository inspection,
normal Git merges, composition proofs, a controlled red/green regression
and complete PostgreSQL/filesystem integration suites. No subagents were
used in this reconciliation.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Codex <noreply@openai.com>
 2026-10-29 10:14:12 -05:00
DottaandPaperclip 27fbdb2a74 perf(workspaces): reuse verified seeds across task branches (#18159) 
## Thinking Path

> - Paperclip manages AI agents and their workspaces.
> - Remote runs stage a recoverable copy of the selected host workspace.
> - New tasks use separate workspace IDs and Git branches.
> - The verified seed cache currently treats those identities as
different content.
> - This causes repeated clones and seed verification when task files
and Git commits are identical.
> - This pull request shares immutable seed content within a company and
applies each task's branch to a private archive.
> - It also removes credential broker calls for two local Git snapshot
commands.

## Linked Issues or Issue Description

**What happened?**

New remote tasks rebuild the same Git seed under different workspace IDs
and branches. In a measured staging run, preparation took 29.7 seconds.
Git clone took 6.1 seconds and optional seed publication took 7.8
seconds. Publication overlapped the 4.4-second transfer, so those values
cannot be added. Local `diff-files` and `write-tree` commands also
request GitHub credentials.

**What did you expect?**

Reuse a verified seed when filesystem contents, Git HEAD, origin, and
nested repository layout match. Preserve each task's branch, recovery
archive, ignored files, and copyback behavior. Keep local snapshot
plumbing independent of GitHub credential capture.

**Steps to reproduce**

1. Configure a remote native environment and a Git project workspace.
2. Create two tasks from the same source commit with different task
branches and workspace IDs.
3. Inspect the workspace preparation spans. Both tasks currently clone
and publish the same seed content.

**Paperclip version and deployment mode**

Reproduced on Core `25f1070fbdabd96b26cdf199e624a77905f3ca37`, with a
managed remote sandbox and native Runner. The local comparison used a
public repository at a fixed commit with roughly 13,000 entries.

Refs #17886, #16398, #16399, #17357. The Git launcher extension builds
on Jamie Buchanan's work in #17886. The other PRs establish the seed
verification, recovery, and eviction contracts preserved here. A search
found no open PR for sharing these verified seeds across task branches.

## What Changed

- Use a company-scoped cache bucket for native host-current preparation.
Keep the existing bucket and company storage limits and eviction rules.
- Remove branch names from the shared generation identity. Keep file
contents, Git HEAD, origin, and nested layout in that identity. Keep the
default workspace-scoped identity unchanged for other callers.
- Store a detached Git archive in the shared cache. Copy it and append
the selected branch metadata before upload and private per-run
persistence. Existing durable-seed replay can read the private archive
without the new cache option.
- Forward the scope through the execution target and both
command-runtime paths.
- Treat `diff-files` and `write-tree` as local Git commands. Network
commands, unknown commands, and managed commit identity keep their
existing credential path.
- Test new workspace IDs, named and detached branches, nested
repositories, dirty and ignored files, symlinks, recovery, corrupt
caches, refused publication, and copyback.

## Verification

- Current head `6087bac274d50804d1b332646e43f3b8d41d798f`: 53 checks
passed, two optional checks skipped, Greptile 5/5, no unresolved review
threads. Linux CI passed.
- Adapter-utils TypeScript 7.0.2 typecheck passed.
- Server package typecheck passed, including its Runner and plugin build
prerequisites.
- 140 seed-cache, seed-reuse, and sandbox-runtime tests passed after
rebasing.
- 44 Git launcher and command-runtime tests passed after rebasing.
- 30 native workspace preparation, warm history, and
retention-configuration tests passed. All 56 retention tests also passed
(5.72 seconds) after repairing missing library aliases inside the
installed macOS Postgres package. That local dependency repair changed
no repository source. The earlier startup timeouts occurred before test
assertions.
- The new seed reuse regression failed before the change because the
second branch cloned again. It passes through the execution target and
both command-runtime paths after the change.
- Clean local preparation comparison: two alternating measured pairs,
with an existing verified company seed and new task branches. Baseline:
19.63 and 9.85 seconds. Shared cache: 5.18 and 4.80 seconds. Median:
14.74 to 4.99 seconds. Each baseline cloned and published; each
shared-cache run skipped both. Branch and file checks passed. The
complete comparison loop took 58 seconds.
- The comparison used real Git, archives, snapshot workers, and local
file transport. It used no model calls. It does not establish a complete
staging response time. The first generation still pays the cold
preparation cost.

## Risks

- A shared cache hit requires an identical generation that is still
present. The company bucket retains the existing 16-generation/1 GiB
bucket limit; busy companies may miss more often.
- The branch binding appends entries to a private tar archive. Tests
cover the host platform; Linux CI must also pass before merge.
- A cache miss, corruption, or capacity refusal must retain the private
recovery path. Existing tests exercise those cases.
- Git plumbing can still encounter repository-specific filters or
missing objects. This extends the existing local-command policy by two
commands; it does not grant network credentials to them.
- The latest measured real staging reply took 77.8 seconds. This change
targets a bounded part of that path and does not claim the
under-15-second response goal is met.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning and repository tools. The
session does not expose the exact model revision or context window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-29 09:19:23 -05:00
DottaandPaperclip 7e27bc5007 fix(tasks): keep anchored opens compact and stabilize saved cards (#18092) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Task pages must make the current instruction and recent conversation
readable quickly.
> - Compact startup avoids reading all earlier messages and run
activity.
> - Every URL fragment previously selected the full-history path, even
for documents.
> - Saved document cards also arrived after the conversation was
visible. PR statistics could add another row later.
> - This pull request keeps anchors compact and reveals the saved
conversation structure together.
> - The benefit is stable reading position without loading unrelated
history.

## Linked Issues or Issue Description

Related: #17593 introduced compact startup. #14667 covers earlier reveal
and scroll work. This fix retains the bounded history contract.

**What happened?**

Opening a task document link loaded all task history. Ordinary cold
opens could reveal messages before saved document cards and plans
arrived. Later PR metadata added a statistics row and changed card
height.

**Expected behavior**

Every URL fragment keeps compact startup. Document links open their
pane. Older-comment links fetch only the target comment. Saved cards
keep their geometry after first reveal.

**Steps to reproduce**

1. Create a task with many comments, a saved plan, documents and saved
PR work products.
2. Delay the enriched task response and open the task. Observe cards
inserted after messages appear.
3. Open a document fragment. Observe full comment, activity and run
reads.
4. Delay PR metadata refresh. Observe each card grow when diff counts
arrive.

## What Changed

- Remove fragment and tab checks from automatic history expansion.
Resolve validated comment UUIDs through the authorized single-comment
endpoint, encode their path components, retain fetched targets across
document navigation, and preserve cancellation/deletion cache updates.
- Include the saved plan, document metadata and local work-product
projection in the core task read. Skip recovery, workspace and
related-task enrichment. Do not refresh providers. Summary queries no
longer select other document bodies.
- Seed saved work-product and plan queries from core, including already
pending or errored reads, without replacing known lists or an
authoritative empty result. Coordinate the bounded recent reads, saved
cards, latest transcript and settled initial pane width before first
reveal. Keep the thread and composer mounted during refresh and explicit
expansion.
- Open continuation summaries in the document pane. Reserve PR
statistics geometry while provider metadata loads.
- Add a reusable synthetic local task, nine browser checks, frame
measurements and video. Extend the PostgreSQL core-route fixture with a
saved PR row and assert its identity, URL, state and stored metadata
survive the response. Update design guidance and capability maps.

## Verification

- Red: visible document/plan cards changed from 0 to 1 to 4; thread
height grew from 690 to 1,432 pixels. Green: all seven saved cards
appear before reveal, with constant thread height and zero per-card
movement in the final reviewed browser run. The final 1,492-pixel layout
includes the reserved PR statistics lines.
- Red: delayed PR diff counts changed all three cards from 62 to 82
pixels. Green: their heights stay at 82 pixels before and after metadata
refresh.
- Nine browser journeys pass: delayed enrichment,
document/comment/unknown/continuation anchors, document clicks and
explicit history expansion with a retained draft, missing-comment retry,
delayed PR statistics, and retained older-comment position after
document navigation.
- 583 focused UI tests passed on `c91fa107ba`. After the plan-cache
follow-up, all 212 affected page/cache/plan-hook tests and UI typecheck
pass, including 14 plan-hook cases. The original 16 focused PostgreSQL
tests passed for the server changes.
- Merge-review red: holding the duplicate stored work-products read
blocked the first reveal past 10 seconds despite core already containing
the saved cards. Both empty and populated core projections now reveal
while that request remains pending. Malformed-comment tests prove no
target request is made; API tests prove path encoding and abort-signal
forwarding.
- The populated core-route test now inserts a real saved PR work product
alongside the plan and documents, then checks the complete saved
projection through HTTP. All ten task cold-open PostgreSQL tests pass.
- Plan-cache review red: an existing pending or errored plan query
prevented the saved core plan from reaching the consumer. Four real-hook
cases failed before the data-presence fix and pass afterward, for both
present and absent plans. A separately fetched newer plan or known
absent plan remains authoritative.
- Further layout red: the initial properties pane reduced visible thread
width from 775 to 742 pixels after reveal. The reveal now waits for two
stable width frames; the browser check includes width, height, card
counts and per-card geometry.
- `pnpm -r typecheck` and `pnpm build` passed for the original change;
UI typecheck and build pass after review fixes. Token, module-boundary
and map checks pass. `pnpm test:run` was stopped after about 15 minutes
in the serialized server lane. No full local test-suite pass is claimed.
On the earlier head, hosted build/typecheck and all Vitest lanes passed;
one E2E shard failed because the agent-configuration clipboard helper
copied page text into its YAML parser. On `c91fa107ba`, Greptile was 5/5
and every E2E shard passed, including the earlier clipboard case. One
unchanged Discord chat-binding test in server shard 8 returned false
after 185 ms; its 16-test integration file passes locally without
changes. The CI cause remains unestablished. On `be436ce492`, both
server shard 8 and E2E shard 7 passed without changing those tests or
their product code. On final head `e357b3bced`, all CI jobs pass,
including full general and serialized tests, all eight E2E shards,
typecheck, build and the aggregate verification gate. Exact-head
Greptile is 5/5 with no findings. The overall status rollup contains 55
successes, two intentionally skipped Storybook checks and a neutral
Paperclip Review. That advisory review ended without a complete
assessment; all previously posted findings are resolved and it added no
current findings.
- Test startup exceeded an initial 5-second deadline once while
development assets and other checks were active. The final browser
checks allow 10 seconds. These runs establish geometry and bounded
requests; they do not establish a reliable latency improvement. An
initial 503 error fixture exercised global reconnect/backoff. The
target-comment failure test uses a terminal 404.
- Browser data is synthetic. Provider metadata is simulated for the
delayed-response check. No staging deployment or live provider execution
is claimed.

## Risks

- Core task reads now include local saved-card queries and the saved
plan body. Other document bodies and all historical run activity remain
outside that path.
- First reveal waits for the bounded current structure. The existing
15-second incomplete-history fallback and inline retry remain available.
- Continuation-summary links open in the document pane in the current
task interface. Classic tasks keep their existing surface.
- No migration or mutation permission change.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, code editing, shell tools
and browser testing. The exact serving model ID and context-window size
are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-29 09:13:01 -05:00
DottaandPaperclip 2a6d4d1694 perf(db): index canonical task activity lookups (#18043) 
<!-- entire-trail-link-start -->
https://entire.com/gh/paperclipai/-/trails/177/changes/paperclip/177
<!-- entire-trail-link-end -->

## Thinking Path

> - Paperclip helps people manage AI agents and their tasks.
> - Task lists rank tasks by their latest canonical activity.
> - The existing lookup reads every event for each task before
pagination.
> - Query-only alternatives failed a narrow visibility control or the
performance goal.
> - This PR adds an index that lets the unchanged lookup stop at the
newest qualifying event.
> - The result is less history work with the same permissions, ordering,
and response.

## Linked Issues or Issue Description

Refs: #13803. That broader proposal stores a new activity field with
triggers and a backfill. This index was authored independently and uses
the existing query.

**Subsystem affected**

Task-list activity ranking and the database schema.

**Current behavior**

The list computes `MAX(activity_log.created_at)` for every eligible
task. The existing entity index reads 50 events per task across 1,199
candidates in the representative fixture.

**Proposed behavior**

Add `(company_id, entity_type, entity_id, created_at)`. Keep the
production task service and permission helpers unchanged. PostgreSQL
reads backward until an event passes the existing exclusions.

**Reason and benefit**

Fresh paired measurements on the current base improve the
full-description scoped SQL from **44.60 to 20.33 ms (2.19×)**. Activity
buffer accesses fall from **63,547 to 4,796**, and history rows per
probe fall from 50 to one. The complete service median falls from 50.63
to 31.31 ms. Complete positional SQL results and service responses
match.

## What Changed

- Add generated migration `0414_closed_edwin_jarvis.sql` and the
matching Drizzle index.
- Use `CREATE INDEX CONCURRENTLY IF NOT EXISTS` through the existing
top-level runner and invalid-index retry cleanup. The client change is
one whitelist entry.
- Preserve all upstream SQL and journal entries; retain snapshots
0410–0414.
- Cover full and index projections, canonical ordering, invalid-index
recovery, concurrent migrators, and incomplete fixture rejection.
- Repair benchmark seeding, apply requested work memory to both
connections, and record both settings. Hosted defaults now include six
Slack conversations, 642 GitHub conversations, and 53 removed
conversations.

## Verification

- **All final-head CI gates passed**, including full workspace
typechecks, tests, build, Runner checks, and all eight E2E shards.
Greptile is **5/5** on `67ca7ae1cf15fc054ef515ffd8205b1fe30a0d55`; no
unresolved review threads remain.
- Local database/server typechecks and migration numbering/safety checks
pass. Three server regression tests and three migration regression
suites pass on real PostgreSQL.
- The original pre-index work guard fails at eight rows per probe; the
indexed guard now checks both the full-description query and index
projection. The service oracle covers cleanup/inbox exclusions, raw text
IDs, company scope, comments, ties, sort directions, visibility before
pagination, empty lists, and one child per parent.
- The documented pre-index seed command reproduced the review failure
before the fix and now succeeds. A paired run with non-default memory
confirms both independent connections use `16MB`.
- Fresh proof pins pre-index base
`929e840595c8933ca485ac7a2e99413e1e11ca64`, with identical cloned data,
unchanged generated SQL and parameters, two warmups, and five
alternating samples per arm. All **18 comparison cells** preserve every
positional row and complete service response. The 15-case default-memory
matrix passes its physical-work, speed, and control bounds.
- The one-visible-task control is 6.64 → 7.25 ms SQL; the 20-task
control is 3.71 → 3.72 ms. Both pass the declared 25% or 2 ms budget.
Full-query memory controls use matching `4MB` or `16MB` settings.
- Committed 1,000-row write batches measured 40.09 → 43.17 ms median
(+7.7%). Ranges overlap (34.67–71.85 and 34.27–48.05 ms); this is an
observed local cost, not a precise throughput guarantee.
- The final-numbered migration builds a valid index through the normal
runner on the fresh 300,000-row fixture in 0.55 s. Earlier large-fixture
proof of the same index SQL used 5,000 tasks and 1.25M activity rows:
2.58 s first apply, concurrent write/read probes 5.16/2.43 ms,
interrupted invalid-index cleanup, successful retry, and one receipt.
Current migration recovery tests verify the renumbered path.
- [The proof
document](https://github.com/paperclipai/paperclip/blob/67ca7ae1cf15fc054ef515ffd8205b1fe30a0d55/docs/performance/task-list-activity-index.md)
retains the historical baseline, rejected rewrites, and the earlier
matrix invalidated by severe host contention. Those historical timings
are not substituted for the fresh measurements above.

## Risks

The migration is an exception after measured query-only alternatives
failed. The index adds storage and work to every activity write: about
30 MiB for 300,000 rows and 123 MiB for 1.25M rows. Concurrent creation
avoids the writer blocking observed with an ordinary index build, but
long transactions can delay it. Local timing does not establish Neon
build duration.

The query still ranks all eligible tasks, and excluded newer events
still need to be skipped. These are local SQL/service measurements;
deployed HTTP latency remains unverified. No hosted DDL or deployment
was performed.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, code execution, and tool
use. The environment does not expose the exact model identifier or
context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either linked existing issues or described the issue using
the relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links
- [x] My branch name describes the change and contains no internal issue
id
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-29 09:06:41 -05:00
DottaPaperclipCodex
 		ba07f85cad perf: schedule agent timers from durable deadlines (#17876) 
<!-- entire-trail-link-start -->
https://entire.com/gh/paperclipai/-/trails/58/changes/paperclip/58
<!-- entire-trail-link-end -->

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agent timers must start due work and preserve queue admission rules.
> - The server woke their scan on every heartbeat interval, even with no
eligible timer.
> - Idle scans keep the database and instance active.
> - This pull request derives the next deadline from saved policy and
work records.
> - Supported producer transactions notify the shared coordinator when
eligibility changes.
> - An idle timer worker stops scanning and resumes when real work or a
deadline requires it.

## Linked Issues or Issue Description

Refs #17863. This is slice 63 of the recurring-work migration and
follows the merged issue-monitor deadline PR.

Related: #16632 changes timer issue scope; #16881 changes live-run
eligibility. This PR preserves the current admission policy. Those
policy changes are separate work. A future time-sensitive predicate must
expose its next transition deadline.

**What existing behavior does this improve?**

Agent timer heartbeat scheduling in the server.

**Current behavior**

A recurring heartbeat wakes the timer scan even when companies are
paused, timers are disabled, or no actionable task exists.

**Proposed behavior**

Register the earliest eligible interval or positive daily-cap reset
deadline. Re-read on supported producer commits and startup. Leave no
recurring database scan when no timer obligation remains.

## What Changed

- Keep policy and deadline calculation in the heartbeat queue.
Scheduling retains company, reporting-chain, worktree, and CAS gates.
- Use the existing deadline coordinator. Remove only the timer wake from
the server heartbeat loop.
- Notify before existing transaction writes that change eligibility or
deadlines. Cover configuration, lifecycle, company budgets, task
ownership, recovery, conversations, accounting corrections, and
completion.
- Use the lifecycle status-change hook for saved hire rejection. Cover
checkout's second adoption transaction after an initial failed CAS.
- Preserve pending work through idle and maintenance drains. Check
cancellation between due agents. Reconstruct on restart.
- Document the supported writer model and producer inventory. Add no
schema, table, or provider branch.

## Verification

- **1,710 passing cases across 49 complete files**, no skips, on
accepted source `32db86c1974dcbf794f01a8c5e42024e2ad071fc`. Coverage
includes real timer policy/CAS/producers, monitor and shared deadline
consumers, native reconstruction, Slack lifecycle,
skills/settings/configuration, queue isolation and agent lifecycle.
- Database build, production server typechecking/build, module
boundaries, target migration order and Runner profile identity pass.
Twenty-nine fresh PostgreSQL 17 schemas plus per-file fixtures applied
all 412 journal entries through index 413; all selected source hashes
and temporary fixture imports were restored exactly.
- Four new deadline composition cases verify future/overdue timing,
minimum scan spacing and disarming. A controlled mutation dropping only
the inherited continuation option fails; the composed source passes with
identical assertions and timers.
- Normal parent reconciliation preserves task-drain expiry and both
deadline contracts. The final monitor-parent publication changes only
its audit, preserving every qualified runtime and selected test source.
- Historical failures and review corrections remain in [the timer
audit](doc/idle-agent-timers.md). No full local repository-suite, live
provider, measured suspend/wake, external-alarm or fleet-savings claim.
Final-head CI/security, exact-head Greptile 5/5 and actionable Ponytail
findings remain separate merge gates.

## Risks

- Process-local notifications cover supported service/API writers in one
server process. Direct SQL and independent replicas need an explicit
ingress contract. Startup reconstruction is not live cross-process
notification.
- Notifications can batch for up to the configured heartbeat interval.
Known deadlines retain their scheduler path; failed reconciliation
retains bounded retry.
- Enabled timers without an actionable-work requirement remain real
recurring obligations. Zero caps require policy changes; positive
reached caps use UTC midnight.
- The timer fixture substitutes adapter enqueue. Existing admission
tests also pass, but this is not deployed provider or sleep
qualification.
- Cloud does not gain legacy agent timer alarms in this PR. Existing
sleep-safety blockers remain. No database/host suspension or fleet
savings claim is made.
- Parent-stack CI and review must pass before merge. This PR does not
authorize merging its dependencies.

## Model Used

OpenAI Codex (GPT-6-based coding agent), using repository inspection,
normal Git merges, composition proofs, a controlled red/green regression
and complete PostgreSQL integration suites. No subagents were used in
this reconciliation.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Codex <noreply@openai.com>
 2026-10-29 08:58:27 -05:00
DottaandPaperclip b81b870732 Add QA smoke journeys and OpenCode run traces (#18098) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Product E2E journeys verify saved application behavior across
deployments.
> - Task creation and questions need reusable smoke selections and exact
run attribution.
> - Operators also need to inspect the OpenCode execution behind a
measured question.
> - Company observability does not currently support native OpenCode.
> - This change adds the journeys and a bounded projection of normalized
OpenCode run events.
> - The projection follows the configured capture policy and retains the
current run identity across warm reuse.

## Linked Issues or Issue Description

Related foundation: #17262. Related alternative: #17989 exports metadata
from OpenCode's native OTLP stream. This change uses normalized run
events to support the existing content switches and per-run linkage. It
does not forward the native stream. The two PRs overlap in launch
registration and must be reconciled before both land.

**What existing behavior does this improve?**

Reusable Product E2E smoke selections and company observability for
native OpenCode.

**Current behavior**

The journey catalog lacks reusable task creation and attached-agent
question checks. Native OpenCode is marked unsupported for company
observability, so a successful destination connection test does not
prove that agent traces will appear.

**Proposed behavior**

Add explicit task-create and ask-question selections. Verify saved state
and exact company, issue, agent, model, environment and successful run
attribution. Export normalized OpenCode turns, tools and usage to the
configured OTLP destination. Honor each capture switch and redact known
secrets.

**Reason and benefit**

CI can run a small measured smoke selection, retain its screenshots and
checks, and link results to traces belonging to the same run.

**Breaking changes**

Existing selectors remain unchanged. OpenCode export is opt-in through
existing company settings. Legacy OpenCode stays unsupported.

## What Changed

- Add reusable task creation and question journeys, configured-agent
capability, owned-fixture checkpoints and strict evidence validation.
- Add an OpenCode-owned projection of normalized run events. No provider
SDK or raw payload schema enters heartbeat coordination.
- Extend the private launch union and reserved environment allowlist for
native OpenCode. With export disabled, no receiver request runs.
- Keep destination headers in the parent wrapper and out of the actual
provider child environment and configuration.
- Bound each turn to 64 tool and 64 usage entries, 8 KiB per text field
and a 1 MiB request. Use a three-second request deadline, four
concurrent exports, and no retries or idle worker.
- Pass the rotated runner authority to each proxy turn. A retained
process no longer attributes later turns to its original launch run.
- Update settings copy, observability documentation and capability maps.

## Verification

- `pnpm -r typecheck` and `pnpm build` passed after the implementation
changes.
- Adapter launch/redaction: 9 tests passed. Projection and actual proxy:
12 tests passed before the review fix; the expanded projection suite
passed all 12 cases after the fix. The proxy test exercises two runs in
one process against a real local HTTP receiver and a fake provider.
- Full OpenCode driver/projection suite: 72 passed, one existing
response_wake test timed out at five seconds. A focused check then
passed both existing omitted-continuation cases and the new trace export
case. The timeout cause remains unconfirmed.
- Runner package TypeScript and Rust `cargo check` passed. Map and UI
token checks passed.
- Prior journey validation: 273 tests across 12 files and the journey
TypeScript check passed.
- Broad local `pnpm test:run` encountered native-runtime integration
failures with repeated embedded PostgreSQL initialization failures and
was stopped. All 16 failing cases passed in an isolated run outside the
filesystem sandbox (961 unrelated cases unselected). The broad local run
is not a clean acceptance result. A later focused database-backed
chat/heartbeat run passed all six selected observability cases,
including native OpenCode launch and successful allocation (1,809
unrelated cases were unselected).
- All CI gates passed for `21085f880d8babc9efeb22d0623db99613d33093`,
including packaged-runner, clean-install, browser and build checks:
https://github.com/paperclipai/paperclip/actions/runs/40884658995 .
Greptile reviewed the same commit at 5/5 with no unresolved findings.
- Verified OTLP JSON acceptance and indexing with an explicitly
synthetic transport marker. No live OpenCode model trace or deployed
smoke acceptance is claimed yet.

## Risks

Export is best effort. Abrupt termination, recovered in-flight turns
without a new submitted event, limits or receiver failure can lose
traces. Export failure does not change task success. The projection
represents observed normalized events, not raw native model spans or
hidden reasoning. Known-secret redaction is not an OS isolation
boundary. UI changes can break journey locators. Deployment adapters
retain credential, ownership and cleanup responsibility. Both the
controller and native runner bundle must be updated before live
qualification.

## Model Used

OpenAI Codex, GPT-6, with reasoning, code execution and browser tools.
The exact serving model ID and context size are not exposed to this
session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (targeted suites; broad
local limitation documented above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-29 08:57:12 -05:00
DottaPaperclipCodex
 		0f6f0a3fee fix(server): schedule issue monitors without heartbeat polling (#17863) 
<!-- entire-trail-link-start -->
https://entire.com/gh/paperclipai/-/trails/56/changes/paperclip/56
<!-- entire-trail-link-end -->

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Issue monitors save a future check on an existing issue.
> - The heartbeat timer still scans monitors on every tick, even when
none are due.
> - The saved deadline and claim already provide durable scheduling
state.
> - This pull request gives monitors their own deadline worker and
connects state changes to it.
> - Idle monitor work stops querying. Existing admission and recovery
rules remain in force.

## Linked Issues or Issue Description

Refs #17840, #16325. Related: #17171 addresses rearming a monitor after
its admitted run fails. This PR changes when monitor inspection runs and
follows the merged routine prerequisite.

**What existing behavior does this improve?**

Issue-monitor scheduling and idle database use.

**Current behavior**

Heartbeat ticks call the combined agent and monitor sweep. The existing
next-deadline query only retains future monitors. Polling covers overdue
leftovers, stale claims, and changes that make a saved monitor eligible.

**Proposed behavior**

Use the shared deadline worker. Retain overdue work with bounded
retries. Wake after committed changes to monitor state, issue
eligibility, company state, or native ownership. Keep legacy agent ticks
separate for the next slice.

## What Changed

- Add monitor-specific sweep and deadline methods to the existing
scheduling owner. Keep the manual combined method.
- Include overdue work and the strict five-minute claim fence. Exclude
active native owners until their state changes.
- Register monitors with the shared deadline worker. Heartbeat ticks
wake only the legacy agent timer worker.
- Connect direct issue-state writers and existing execution-control
notifications to monitor work. Checkout uses a short transaction around
its existing CAS. Provider work stays outside it.
- Add real database lifecycle tests and retain routine, startup,
admission, execution-control, and native-projection coverage.
- Record the obligation, producer inventory, recovery behavior, and
limits in `doc/plans/2026-10-27-issue-monitor-sleep-audit.md`.

## Verification

- **1,233 passing cases in 35 complete files**, no skips, on accepted
source `c24aadfc91e9394d73c799f9e05fc67202160557`. Coverage includes
real monitor scheduling/admission, native settlement, committed
producers, retained routines and all shared deadline consumer
integration suites.
- Database build, production server typechecking/build, module
boundaries, target migration order and Runner profile identity pass.
Twenty-four fresh PostgreSQL 17 schemas plus per-file fixtures applied
all 412 journal entries through index 413; every selected source hash
and temporary fixture import was restored exactly.
- Normal parent reconciliation preserves master's admitted-wake result
and scoped issue deletion while keeping committed monitor notifications.
All other paths match the automatic merge; packages remain byte-exact to
the qualified parent. The final routine-parent publication changes only
its audit.
- Historical failures and corrections remain in [the monitor
audit](doc/plans/2026-10-27-issue-monitor-sleep-audit.md). Old
delegated-Slack rate and Runner drain CI failures are retained; the
current inherited clock fixture and package source are byte-identical to
merged #17835. Fresh final-head CI remains required.
- No full local repository-suite, live provider, deployed suspend/wake,
external-alarm, production-load or fleet-savings claim. Final-head
CI/security, exact-head Greptile 5/5 and actionable Ponytail findings
remain separate merge gates.

## Risks

- A missing producer notification can strand a monitor after polling is
removed. The audit covers ordinary writes, direct status/assignment
writers, recovery, and native ownership changes.
- The writer model is process-local. Independent SQL writers and other
replicas need a supported notification or reconciliation contract.
- Overdue work uses the configured heartbeat interval as a retry floor
for the monitor worker. New notifications during that backoff wait for
its next pass. Normal state-change bursts coalesce for one second.
- Notification fanout can inspect monitors during other active work. An
unchanged idle subsystem has no recurring monitor scan.
- The legacy agent timer still polls. Cloud sleep proofs and external
alarm coverage are unchanged.

## Model Used

OpenAI Codex (GPT-6-based coding agent), using repository inspection,
ordinary Git merges, composition proofs, PostgreSQL fixtures and
complete local integration suites. No subagents were used in this
reconciliation.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Codex <noreply@openai.com>
 2026-10-29 08:40:56 -05:00
DottaPaperclipCodex
 		929e840595 fix(server): schedule routines without heartbeat polling (#17840) 
<!-- entire-trail-link-start -->
https://entire.com/gh/paperclipai/-/trails/47/changes/paperclip/47
<!-- entire-trail-link-end -->

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Routines create work at saved cron deadlines.
> - The heartbeat still wakes the routine worker when nothing is due.
> - Its old next-deadline query also drops overdue work left by capped
catch-up.
> - Removing the heartbeat alone can strand that work and miss company
reactivation.
> - This pull request uses saved deadlines and committed producer
changes.
> - Idle routine scheduling stops querying until a change, retry, or
real deadline.

## Linked Issues or Issue Description

Refs #16514. Merged prerequisite: #17836. Related Cloud wake contract:
#16215.

**What happened?**

Every heartbeat wakes the routine scheduler. The next-deadline query
excludes overdue triggers. A 27-tick hourly backlog loses its deadline
after the first 25-run pass. Company and budget lifecycle changes also
lack routine schedule signals.

**Expected behavior**

Use the actual next saved deadline. Retain overdue work. Refresh after
supported producer transactions commit. Keep idle schedules quiet
between obligations.

**Steps to reproduce**

1. Create an hourly routine with capped catch-up and 27 overdue ticks.
2. Run the registered worker once. The base processes 25 ticks and
reports no next wake.
3. Create a future schedule and pause or reactivate its company. The
base does not refresh its registered deadline until another wake.

## What Changed

- Remove only the routine worker wake from the heartbeat tick.
- Put routine eligibility and the next-deadline read in
`routineService`. Include overdue triggers. Keep trigger/routine company
scope consistent and require an active company.
- Reuse `registerDeadlineWork` for retries, request generations, and
passive drain behavior. Narrow its coordinator type to the `register`
method it uses. Existing consumers keep the same runtime behavior.
- Signal routine work from manual schedule invocation, company
pause/resume/archive/delete, and budget-induced company pause/resume.
Existing routine and trigger producers retain their transaction signals.
- Stop between trigger batches during drain/shutdown. Finish a claimed
batch so its advanced timestamp does not discard admitted work.
- Add real PostgreSQL lifecycle tests and extend startup/query fixtures.
Add a validated external database option to the company suite, including
migrations and client closure.
- Record producer coverage and limits in
`doc/plans/2026-10-27-routine-schedule-sleep.md`.

## Verification

- **417 passing unique cases across 14 complete files**, no skips, on
accepted source `4659ef76d398193b15e7812d4a97f3da3b743226`. Coverage
includes the real deadline worker, database eligibility queries, routine
services/HTTP/routes/plugins, budget/company producers, git/device
fixtures, startup and shared coordinator.
- The first combined run passed 416 cases and had one UUID HTTP
authorization timeout at its unchanged 20-second limit. One fresh
complete 13-case E2E diagnostic passed (the affected case took 425 ms).
No assertions, timeouts or production source were changed; the initial
timeout's cause is not established.
- Database build, production server typechecking/build, module
boundaries, target migration order and Runner profile identity pass.
Fresh PostgreSQL 17 fixtures applied all 412 journal entries through
index 413; all selected test hashes and temporary fixture imports were
restored exactly.
- Normal parent merges preserve the tested production and test source.
Historical acceptance and review corrections remain in [the routine
audit](doc/plans/2026-10-27-routine-schedule-sleep.md).
- No full local repository-suite, deployed suspend/wake, external alarm,
live provider or fleet-cost claim. Final-head CI, security, exact-head
Greptile 5/5 and actionable Ponytail findings remain separate merge
gates.

## Risks

- Company pause now prevents due routine claims, matching the prior
deadline query. Rows remain saved for reactivation. Existing routine
reactivation, project-pause skip records, cron, timezone, catch-up, and
concurrency policies remain with the routine owner.
- Signals cover supported service writers in one process. Independent
SQL writers and other replicas need their own integration.
- This proves worker quiet, not host/database suspension or fleet
savings. Cloud idle proof and external wake coverage are unchanged.
- Issue-monitor and legacy-agent-timer polling remain separate duties.
This slice does not finish the full migration stack or its CI/review
work.

## Model Used

OpenAI Codex (GPT-6-based coding agent), using repository inspection,
ordinary Git merges, source-composition proofs, PostgreSQL fixtures and
local lifecycle tests. No subagents were used in the current
reconciliation.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Codex <noreply@openai.com>
 2026-10-29 08:08:04 -05:00
DottaPaperclipCodex
 		fa5e70faf4 fix(server): schedule automatic backups through shared lifecycle (#17836) 
<!-- entire-trail-link-start -->
https://entire.com/gh/paperclipai/-/trails/39/changes/paperclip/39
<!-- entire-trail-link-end -->

## Thinking Path

> - Paperclip manages agent work and its durable state.
> - Automatic database backups protect that state on a configured
schedule.
> - Their independent interval is outside the shared deadline and
shutdown lifecycle.
> - An enabled backup schedule owes real work even when no queue
changed.
> - The existing coordinator can own this deadline without another table
or notification protocol.
> - This change preserves backup policy and moves scheduling into that
shared lifecycle.

## Linked Issues or Issue Description

Refs: #17835. Related work: #11756 (abortable backups), #3549
(stale-backup catch-up), #2120 (interval unref), #16364 (bounded
shutdown steps). This slice changes scheduling; it does not implement
those separate features.

**What existing behavior does this improve?**

Automatic database backup scheduling and shutdown ownership.

**Current behavior**

A standalone interval invokes backups, skips drain and warm standby, and
is not registered with the shared work scheduler. Its future deadline is
invisible there. The interval is not cleared or joined by the delivery
coordinator during shutdown.

**Proposed behavior**

Register the configured backup deadline with the shared coordinator.
Call the existing backup owner only when due. Keep the startup-relative
phase, skip missed slots, retain the next interval after failure or a
busy-owner skip, and join active scheduled work on shutdown.

**Reason and benefit**

One scheduling lifecycle owns overlap, admission, deadlines and
shutdown. Waiting does not query the database. This is a true periodic
obligation, so enabled backups still recur; it is not an empty-queue
polling removal or a claim of measured savings.

This existing slice now follows the merged import-spool parent #17835
and targets master. Merge authorization and exact-head CI/review gates
are tracked separately.

## What Changed

- Add a small backup scheduling owner using the existing coordinator and
scheduler.
- Replace the automatic-backup interval in startup. Register only when
backups are enabled, independently of heartbeat scheduling.
- Preserve the existing backup executor, manual serialization, retention
reads, pre-sleep verification, restart marker and host-sleep gates.
- Test deadlines, drain/standby, failures, busy-owner skips, slow
attempts, restart and shutdown. Update startup-marker tests to exercise
the registered deadline.
- Record the audit and update inventory item 15. No schema or backup
format changes.

## Verification

- Current source acceptance: **130 passing cases in four complete
files**, no skips: backup worker, startup checkpoints, scoped deadline
work and delivery coordinator. Database build, production server
typechecking/build, module boundaries, target migration order and Runner
profile identity pass.
- Runtime and all four selected test sources are byte-identical to
accepted source `11775eee7ed795b989cbb31052e69643c468c3cf` after normal
parent reconciliation. The parent carries the tool-access test-order
repair; it changes no backup behavior.
- Historical broader checks and the real JavaScript backup/restore
probe, including their intermediate failures and limits, remain in [the
backup audit](doc/plans/2026-10-26-database-backup-sleep-schedule.md).
They are not claimed as repeated current-head acceptance.
- No full local repository-suite, deployed suspend/wake, pg_dump, live
provider or cost-savings claim. Final-head CI, security, Greptile 5/5
and any actionable Ponytail findings remain separate merge gates.

## Risks

- Enabled backups still require periodic database and disk work. This
change centralizes the deadline; it does not reduce backup frequency or
establish deployed cost savings.
- The backup engine has no AbortSignal contract. Coordinator stop joins
an active scheduled backup, so a stalled backup can delay graceful
shutdown. Bounded backup cancellation remains separate work.
- Restart still starts a new ordinary interval; it does not replay
historical slots. The existing pre-sleep marker still requests an
immediate verified restart backup and persists on failure.
- A due backup waits during drain or standby and retries on the next
coordinator attempt after admission reopens, within one configured
interval.
- The internal deadline is not a persisted Cloud alarm. Existing
scheduled-backup and checkpoint sleep gates remain unchanged. The real
probe qualifies the JavaScript engine only, not pg_dump or deployed
shutdown latency.

## Model Used

OpenAI Codex (GPT-6-based coding agent). Repository inspection, ordinary
Git merges, source-composition proofs and local lifecycle tests were
used for this reconciliation. No subagents were used in the current
merge campaign.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes:` / `Closes:`
/ `Refs:` OR (b) described the issue in-PR following the relevant issue
template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub URLs)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Codex <noreply@openai.com>
 2026-10-29 07:55:53 -05:00
DottaPaperclipCodex
 		1bceae0c91 Make import spool cleanup wake on changes and saved deadlines (#17835) 
<!-- entire-trail-link-start -->
https://entire.com/gh/paperclipai/-/trails/38/changes/paperclip/38
<!-- entire-trail-link-end -->

## Thinking Path

> - Paperclip manages agent work and the files that support it.
> - Resumable company imports keep uploaded parts until apply or expiry.
> - An hourly cleanup loop reads the database for every retained upload
directory.
> - Failed and applying imports can retain files without owing periodic
cleanup.
> - Existing transfer rows and file timestamps already identify real
deadlines.
> - This change schedules those deadlines and wakes cleanup after
relevant changes.

## Linked Issues or Issue Description

Refs: #17801.

**What problem does this solve?**

The hourly import sweep keeps querying retained uploads when no cleanup
is due. It also treats a failed database read as a missing transfer row.
That can cause deletion of a live upload whose directory is old.

**What is the proposed solution?**

Use the existing delivery coordinator with per-transfer deadlines. Keep
commit-aware notifications in the transfer ledger service. Keep
filesystem change notifications in the spool owner. Recover pending
cleanup from existing files and rows at startup.

**What alternatives have you considered?**

A slower periodic sweep would still query idle history. A new job table
would duplicate the existing ledger and directory timestamps. Retrying
only after successful uploads would miss partial file writes that leave
cleanup work.

This is slice 59. Parent #17801 has merged. This publication targets
master and preserves that landed prerequisite. Fresh current-head CI and
review remain required.

## What Changed

- Repair an inherited CI concurrency test. Observe exact PostgreSQL
blockers instead of assuming lock order from two 50 ms sleeps. Keep
production authorization and all fail-closed assertions unchanged.

- Refresh verified-upload activity before writing bytes. Preserve the
guarded cleanup claim without recording a part receipt before the file
exists.

- Isolate unreadable child directories so healthy cleanup continues. Use
unique, self-cleaning route fixtures for external database reruns.

- Replace the hourly timer with the existing scoped deadline worker. No
new table or independent scheduler.
- Schedule open uploads at saved activity time plus 24 hours, preserving
the strict cutoff and guarded cancellation before deletion.
- Keep failed and applying imports quiet while retaining their files.
Clean completed/cancelled leftovers immediately.
- Register run-scoped write intent before every supported ledger
mutation, including nested transactions. The helper stays inside the
existing ledger owner.
- Wake from filesystem changes, including failed writes and removals.
Reconstruct orphan deadlines from directory timestamps after restart.
- Propagate database and permission errors. Only a missing directory
proves absence. Read only the four ledger fields needed for cleanup.
- Preserve startup-only stranded-apply recovery. The cleanup worker
cannot fail live applies. Remove its filesystem subscription and join
active cleanup on shutdown.
- Record producer coverage and limits in
`doc/plans/2026-10-26-import-spool-cleanup-sleep.md`.

## Verification

Current repair source: `5d974e5eb28fc3e28b5e75e36fb18c61582348d5`. The
publication adds a test-fixture repair and audit text to the previously
tested import-spool source; runtime code is unchanged. **765 distinct
cases pass across nine complete files:** the original 293 consumer cases
remain hash-identical, and the full tool-access file passes all 472 in
20.86 seconds.

Controlled evidence: a 500 ms cleanup-query delay fails all three old
timing cases; observed lock barriers pass all three with the same delay
and unchanged policy assertions. The artificial delay is not committed.
Current-head CI's original audience-race failure is preserved. The first
local full-file run also retains one separate 15-second stdio HTTP
timeout; one fresh diagnostic full run passes all 472, without
establishing that timeout's cause or changing its limit. Fresh CI and
reviews are required on this new publication head.

Accepted production source: `87549234b125464dc7fa216227fc9bfa1d93f3d7`.
Publication head: `016b3af51a734ad0ddf574de701679a7bdc33d06`. Adopting
the actual parent squash preserves the complete accepted child tree. The
subsequent changes are the CI test fixture and its audit text.

- **293 cases pass across eight complete server files, with zero
skips:** spool worker 18; HTTP import transfers 25; transfer ledger 9;
cloud floor 8; portability 114; startup 92; coordinator 26; scoped
deadlines 1. Runtime: 12.28 seconds.
- Real filesystem and PostgreSQL cases cover zero idle SQL/directory
scans, actual deadlines, first/concurrent writes, nested
commit/rollback, restart, failure, drain/resume, root and symlink
boundaries, and joined shutdown. Three fresh PostgreSQL 17 fixtures and
isolated PAPERCLIP_HOME keep the run separate. All eight source hashes
remain exact; no assertion, timeout, or source adaptation.
- **Static gates pass:** database build; server typecheck/build; module
boundaries; migration order; profile identity. All 412 migration journal
entries are present through index/tag 0413_tired_kronos.
- **Composition:** 1,237 parent-only and eight owned-only paths remain
exact; 11,521 identical paths and two automatic shared production
results are preserved. The only manual conflict is the status ledger.
Packages and other native worker services equal the parent. Its 4,192
selected cases are prior source-scoped evidence, not added to this
count.
- **Historical failures remain recorded:** earlier trusted-runner and
Greptile service errors are not current passes. The older Ponytail Ship
verdict names another head. Fresh CI, security, and exact-head Greptile
5/5 must pass before merge.
- [Master integration and spool acceptance
audit](https://github.com/paperclipai/paperclip/blob/016b3af51a734ad0ddf574de701679a7bdc33d06/doc/plans/2026-10-26-import-spool-cleanup-sleep.md#current-master-integration-and-spool-acceptance-2026-10-29).
No full local repository, live provider, or deployed sleep claim is
made.

<details><summary>Earlier revision verification and failure
history</summary>

Receiver failure fairness: a controlled 1,100 ms batch reproduced a
30-second delay for healthy due receivers. The worker now supplies its
existing per-item deadline through `nextRunAtOnError`. All 20 receiver
lifecycle tests pass on the owner and combined tip
`31c12f1d68ff4a51e06fc2be2abf2c2aee9f9d3a`; server production
typechecking passes on the owner. The failed receiver retains its own
backoff, and settled receivers return to zero recurring SQL. The
five-second assertion budget is unchanged. See [the owning
audit](https://github.com/paperclipai/paperclip/blob/7435a19f6d18f14466175d72b3b7d893ed39acd0/doc/plans/2026-10-25-chat-receiver-discovery-sleep-audit.md#healthy-receiver-progress-after-a-slow-failed-batch).
Current-head CI and review remain required.

Approved delegated Test-tab calls now evaluate caller and target
policies in the existing claim transaction. Real HTTP regressions first
showed a missing caller charge and shared-limit self-rejection. All 78
connection-management cases and 92 gateway cases now pass on the owning
branch, along with server production typechecking. All 78
connection-management cases also pass at combined tip
`d25f830db89c1623bff3547bfc68015c1c66e0cf`. The three new cases cover
shared charges, separate charges, rollback on denial, duplicate
approval, and rejection of a later authorized call. Intermediate
counter/identity fixture failures and a reused-database Slack collision
are recorded in [the owning
audit](https://github.com/paperclipai/paperclip/blob/1509a56a97f2ffb2ab59a9a27f6d7be16ab46569/doc/idle-approved-tool-recovery.md#delegated-test-tab-admission-after-policy-changes).
The earlier combined checks below remain evidence for their stated
revisions. Current-head CI and review remain required.

Inherited voice-cleanup verification: the controlled slow-provider
regression and all 13 lifecycle tests passed on its owner and combined
tip `ab22d8ed0c09028315607414b83b375e5cc0a60a`. That repair changed only
the fixture clock advancement and its audit note.

Current head: `7435a19f6d18f14466175d72b3b7d893ed39acd0`. This remains a
single-duty PR stacked on #17801.

- Integrated pinned master `8548173fc9efdc0fa07644fb92dcc0b6851a95e5`
with normal merges. Every branch passed generated-schema checks and its
target-branch migration-order gate. Existing SQL is unchanged; the 25
stack migrations follow master’s new migration.
- Duty-specific producer coverage, lifecycle tests, and earlier failure
records:
[2026-10-24-recurring-server-sleep-stack.md](https://github.com/paperclipai/paperclip/blob/47ba0a93c4b01484ad58d16bbeb5a372c76b5a0d/doc/plans/2026-10-24-recurring-server-sleep-stack.md),
[2026-10-26-import-spool-cleanup-sleep.md](https://github.com/paperclipai/paperclip/blob/47ba0a93c4b01484ad58d16bbeb5a372c76b5a0d/doc/plans/2026-10-26-import-spool-cleanup-sleep.md).
- Combined tip `51e6cbe67979b55b340ea9f0f434c5b5c682ccd6`: **355
startup/cancellation/recovery/dispatch/preservation tests**, **75
connection-management tests**, and production server typechecking pass.
These combined checks do not mean every intermediate PR independently
ran the full repository suite.
- Real master-to-stack database upgrade passes: **379 unchanged prior
migration receipts plus 25 new migrations**, with the gateway
responsibility fixture preserved.
- Owning-branch checks: 84 runner installation/settlement cases plus
runner typechecking and generated TypeScript/Rust profile agreement; 253
GitHub review workflow cases. Stop-attribution and held-worker readiness
regressions failed before their fixes.
- Failure accounting: the initial combined run passed 349 cases and
failed six idle assertions in a reused preservation database containing
old cancelled runs. Repeating the combined command with a fresh
preservation database passed. Earlier invalid project selection, merge
import/type errors, and the interrupted full local suite are not counted
as passing checks.
- **Latest-head CI and review remain required.** Prior CI findings
include GitHub idle observation, route teardown, attention cleanup,
touch-picker E2E, workspace health, and voice-cleanup timing; local
passes do not erase those failures. No full local repository-suite,
deployed sleep, fleet cost, or live Copilot qualification claim is made.


</details>

## Risks

- Expiry and active apply must keep their guarded ledger claims. A stale
read must never authorize file deletion after a concurrent resume.
- Database and permission failures remain unresolved work. They cannot
be interpreted as missing rows or empty directories.
- Only supported ledger and spool owner writers emit these local
notifications. Independent processes or SQL writers need ingress;
startup recovery alone cannot wake stopped compute.
- Cloud sleep blockers remain. No deployed suspension, external alarm
coverage, live provider qualification, or cost savings is claimed.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, code execution, real
PostgreSQL and filesystem tests, and GitHub tools. The exact runtime
model identifier and context-window size are not exposed in this
session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes:` / `Closes:`
/ `Refs:` OR (b) described the issue in-PR following the relevant issue
template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub URLs)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Codex <noreply@openai.com>
 2026-10-29 07:42:14 -05:00
DottaPaperclipCodex
 		c8b7afee06 fix(server): schedule native environment lease settlement (#17801) 
<!-- entire-trail-link-start -->
https://entire.com/gh/paperclipai/-/trails/24/changes/paperclip/24
<!-- entire-trail-link-end -->

## Thinking Path

> - Paperclip manages AI agents and the resources used by their work.
> - Native run completion has several independent settlement duties.
> - The recurring run reaper repeated environment-release discovery for
completed native history.
> - Existing lease rows already record releases that still need work.
> - This pull request wakes the existing release owners from committed
changes and schedules only unfinished release debt.
> - Completed history becomes quiet while stop evidence, workspace
preservation, and retention policy keep their existing owners.

## Linked Issues or Issue Description

Refs: #17793, #17717.

**What problem does this solve?**

Periodic native finalization replay calls environment release even when
no lease needs work. This adds recurring database traffic to an idle
instance.

**What is the proposed solution?**

Register one scoped worker on the existing delivery coordinator.
Discover work from native finalizations and existing environment leases.
Wake it from native, lease, and settings transactions. Keep the release
owner's saved pending-cleanup deadlines. Keep retries only while a
concrete lease still needs settlement.

**What alternatives have you considered?**

A slower unconditional scan still queries idle history. A new job table
duplicates existing lease truth. Replacing the compound callback would
also change runtime-service and agent-status duties; those remain
separate slices.

This is slice 58. Parent #17793 has merged. This publication targets
master and preserves that landed prerequisite. Fresh current-head checks
and review remain required.

## What Changed

- Keep ordinary company-only finalization hints out of environment
settlement. Prove zero candidate SQL after quiet and preserve real lease
delivery.

- Reuse the existing durable destroy claim for run-owned cleanup,
excluding concurrent attempts and fencing late receipts while preserving
cleanup recovery.

- Scope lease cleanup notifications to their company/run so settlement
does not rediscover the entire backlog after each release.

- Read only `nativeWorkspaceSync` and `workspaceExportRetry` JSON fields
when discovering environment cleanup. Preserve saved run data and
existing release decisions.

- Reuse the environment runtime's release predicate for discovery. Add
no table or independent timer.
- Wake from existing finalization, projection, lifecycle, recovery,
lease, and settings transaction boundaries. Simple lease acquisition now
uses a transaction too.
- Preserve recorded retention policy, pending-cleanup deadlines,
original lease/provider ownership, and current release guards.
- Re-read durable lease state after a release. A logged or swallowed
per-lease failure does not discharge the obligation.
- Delegate only periodic environment release. Preserve default
live/manual behavior, runtime-service release, and agent-status
callbacks.
- Add an owner/producer/lifecycle audit and real database tests.

## Verification

Accepted production source: `9596fa022b57feea026c4728c80bdc36037b804f`.
Publication head: `fb252c8b0e52ce355e7d98db2e392e4831f99c61`. The actual
parent squash is adopted by a normal merge with the complete accepted
child tree unchanged. Only audit text follows.

- **4,172 distinct cases in 64 complete server files pass**, plus **20
selected chat cases** (4,192 selected total). The main batch reports
3,440 passes and 49 skips. Both complete Stop/read files then pass 51 +
6 cases on separate fresh databases through their supported override.
Their eight pure cases are counted once; skipped cases are never counted
as passes. Separate process recovery passes 410; review response passes
273. The other 1,794 chat cases were not run locally.
- **Quiet producer proof:** the controlled unscoped baseline fails with
three candidate SELECTs after three routine company-only hints. The
scoped case performs zero, releases a real eligible committed-run lease,
and returns to quiet. The same test hash spans red and green.
- **Static gates pass:** database build; server typecheck/build; module
boundaries; migration order; current Copilot profile identity. Packages
equal the accepted parent byte for byte.
- **Fresh fixtures:** 39 prepared PostgreSQL 17 databases plus fresh
helper-owned databases. The schema has all 412 journal entries through
index/tag 0413_tired_kronos. All 65 original test hashes restore
exactly; the indirect Stop fixture is also hashed. No assertion or
timeout changes.
- **Preserved records:** original 49 skips and their completed coverage
remain documented. Parent handoff timeout and prior chat-idle failure
records remain in their owning audits. This does not claim full
repository, full chat, live provider, or deployed sleep acceptance.
- [Current integration and native lease
audit](https://github.com/paperclipai/paperclip/blob/fb252c8b0e52ce355e7d98db2e392e4831f99c61/doc/plans/2026-10-26-native-environment-settlement-sleep.md#current-master-integration-and-native-lease-quiet-proof-2026-10-29).
Fresh CI, security, and exact-head Greptile 5/5 must pass before merge.

<details><summary>Earlier revision verification and failure
history</summary>

Receiver failure fairness: a controlled 1,100 ms batch reproduced a
30-second delay for healthy due receivers. The worker now supplies its
existing per-item deadline through `nextRunAtOnError`. All 20 receiver
lifecycle tests pass on the owner and combined tip
`31c12f1d68ff4a51e06fc2be2abf2c2aee9f9d3a`; server production
typechecking passes on the owner. The failed receiver retains its own
backoff, and settled receivers return to zero recurring SQL. The
five-second assertion budget is unchanged. See [the owning
audit](https://github.com/paperclipai/paperclip/blob/6f3a0b8de149fb5620f15909473eb7c4455b9824/doc/plans/2026-10-25-chat-receiver-discovery-sleep-audit.md#healthy-receiver-progress-after-a-slow-failed-batch).
Current-head CI and review remain required.

Approved delegated Test-tab calls now evaluate caller and target
policies in the existing claim transaction. Real HTTP regressions first
showed a missing caller charge and shared-limit self-rejection. All 78
connection-management cases and 92 gateway cases now pass on the owning
branch, along with server production typechecking. All 78
connection-management cases also pass at combined tip
`d25f830db89c1623bff3547bfc68015c1c66e0cf`. The three new cases cover
shared charges, separate charges, rollback on denial, duplicate
approval, and rejection of a later authorized call. Intermediate
counter/identity fixture failures and a reused-database Slack collision
are recorded in [the owning
audit](https://github.com/paperclipai/paperclip/blob/ae9bff1c06e34702f64a399e58b45a657c133bd1/doc/idle-approved-tool-recovery.md#delegated-test-tab-admission-after-policy-changes).
The earlier combined checks below remain evidence for their stated
revisions. Current-head CI and review remain required.

Inherited voice-cleanup verification: the controlled slow-provider
regression and all 13 lifecycle tests passed on its owner and combined
tip `ab22d8ed0c09028315607414b83b375e5cc0a60a`. That repair changed only
the fixture clock advancement and its audit note.

Current head: `6f3a0b8de149fb5620f15909473eb7c4455b9824`. This remains a
single-duty PR stacked on #17793.

- Integrated pinned master `8548173fc9efdc0fa07644fb92dcc0b6851a95e5`
with normal merges. Every branch passed generated-schema checks and its
target-branch migration-order gate. Existing SQL is unchanged; the 25
stack migrations follow master’s new migration.
- Duty-specific producer coverage, lifecycle tests, and earlier failure
records:
[2026-10-24-recurring-server-sleep-stack.md](https://github.com/paperclipai/paperclip/blob/7e619453567e078a7d745f28bfe8e5ede571f282/doc/plans/2026-10-24-recurring-server-sleep-stack.md),
[2026-10-26-native-environment-settlement-sleep.md](https://github.com/paperclipai/paperclip/blob/7e619453567e078a7d745f28bfe8e5ede571f282/doc/plans/2026-10-26-native-environment-settlement-sleep.md).
- Combined tip `51e6cbe67979b55b340ea9f0f434c5b5c682ccd6`: **355
startup/cancellation/recovery/dispatch/preservation tests**, **75
connection-management tests**, and production server typechecking pass.
These combined checks do not mean every intermediate PR independently
ran the full repository suite.
- Real master-to-stack database upgrade passes: **379 unchanged prior
migration receipts plus 25 new migrations**, with the gateway
responsibility fixture preserved.
- Owning-branch checks: 84 runner installation/settlement cases plus
runner typechecking and generated TypeScript/Rust profile agreement; 253
GitHub review workflow cases. Stop-attribution and held-worker readiness
regressions failed before their fixes.
- Failure accounting: the initial combined run passed 349 cases and
failed six idle assertions in a reused preservation database containing
old cancelled runs. Repeating the combined command with a fresh
preservation database passed. Earlier invalid project selection, merge
import/type errors, and the interrupted full local suite are not counted
as passing checks.
- **Latest-head CI and review remain required.** Prior CI findings
include GitHub idle observation, route teardown, attention cleanup,
touch-picker E2E, workspace health, and voice-cleanup timing; local
passes do not erase those failures. No full local repository-suite,
deployed sleep, fleet cost, or live Copilot qualification claim is made.


</details>

## Risks

- Existing lease owners retain destruction claims, provider identity,
stop evidence, retention policy, and pending-cleanup deadlines.
Incorrect changes here can release a live resource or lose a cleanup
obligation.
- Logged or swallowed release failures remain concrete debt. Real
retries must retain their saved floor across unrelated hints.
- In-process signals do not cover independent SQL or other replicas
without ingress.
- This delegates only periodic environment settlement. Runtime-service
and agent-status duties remain separate. Existing Cloud blockers remain;
no deployed sleep, database suspension, external alarm coverage, or cost
savings is claimed.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, code execution, real
PostgreSQL tests, and GitHub tools. The exact runtime model identifier
and context-window size are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes:` / `Closes:`
/ `Refs:` OR (b) described the issue in-PR following the relevant issue
template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub URLs)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Codex <noreply@openai.com>
 2026-10-29 06:37:48 -05:00
DottaPaperclipCodex
 		829f9e7eaf fix(server): settle Muse terminal assignments from committed changes (#17793) 
<!-- entire-trail-link-start -->
https://entire.com/gh/paperclipai/-/trails/12/changes/paperclip/12
<!-- entire-trail-link-end -->

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Its external-agent module owns Muse assignments, operation receipts,
and worker holds.
> - Running assignments and retained terminal history must not cause an
empty maintenance scan every five seconds.
> - Terminal run changes and operation results already have durable
database transactions.
> - This pull request wakes the existing maintenance worker from those
commits and reconstructs work at startup.
> - The worker retries only concrete settlement debt. It retains
qualification deadlines and authority checks.

## Linked Issues or Issue Description

Refs: #17772. This is slice 57. Parent #17772 has merged.
Related: #17000, #17525, #17752, #17758. The publication targets master
and preserves the merged parent.

**What existing behavior does this improve?**

Muse terminal assignment settlement and the existing Muse maintenance
worker.

**Current behavior**

A running assignment can keep maintenance scanning every five seconds.
Terminal reconciliation rewrites unchanged history. Native finalization
also calls that reconciliation during periodic replay. Discovery
excludes a revoked original binding even when its assignment still needs
settlement.

**Proposed behavior**

Committed run changes and operation receipts wake the existing worker
with the original company, run or binding identity. Startup reconstructs
actionable work from existing rows. Maintenance processes bounded
batches and disarms after settlement. Default live and manual callbacks
remain. The periodic callback delegates this one duty.

**Reason and benefit**

Configured but idle instances stop querying for this obligation. Late
receipts, restarts, drain and historical bindings still have recovery
paths. No table or independent timer is added.

**Breaking changes**

No schema, endpoint or provider protocol changes. Explicit manual
reconciliation retains its unbounded default. This does not alter Cloud
sleep admission or claim that all instance work can sleep.

## What Changed

- Add an owner predicate shared by discovery and locked settlement.
Fenced private-worker uncertainty remains held without an automatic
retry loop.
- Preserve original binding generation and company/agent/run identity,
including revoked bindings. Fence only the terminal assignment during
reconciliation.
- Extend existing maintenance with scoped hints, bounded batches,
concrete-debt retry, startup reconstruction and quiet drain behavior.
- Register notifications inside existing terminal, projection,
cancellation, operation-result and send-failure transactions. Keep
qualification and OAuth signals explicitly global when coalesced with
scoped changes.
- Serialize only four domain IDs. Real native-finalizer tests caught and
prevent a circular-object serialization failure at this boundary.
- Use existing restore/worktree suppression and settings reactivation.
Delegate both periodic finalizer branches while retaining resource
callbacks and default live/manual settlement.
- Add the producer audit and regression evidence in
`doc/plans/2026-10-26-muse-terminal-settlement-sleep.md`.

## Verification

Accepted production source: `584015018ed4a9d57d2b0f192b75847503b54e53`.
Publication head: `305014b030d6c5eb891c70bc57036f4192d4fdad`. Normal
adoption of the actual parent squash preserves the complete accepted
child tree. Only audit text, a migration-count clarification, and the
proved Photon test extension follow.

- **3,715 distinct cases in 52 complete server files pass**, plus **20
selected chat cases** (3,735 selected total). The main 49-file batch
passes 3,008; process recovery passes 410; review response passes 273;
the full Photon file passes 24. The other 1,794 chat cases were not run
locally.
- **Quiet producer proof:** moving only the Muse fanout outside its
parent guard produces six candidate SELECTs and fails the regression.
Keeping the guard yields zero and still settles a real terminal run. The
same test hash spans the controlled red/green pair.
- **Photon review evidence:** repeated active callbacks produce zero
settlement hints/candidate SQL. Three concurrent attention recoveries
produce exactly one of each. The existing locked attention guard was
correct; production needed no fix. The parent security check accepted
the resolved finding before merge. This PR keeps the additional
settlement queue in the existing regression.
- **Static checks pass:** database build; server typecheck/build; module
boundaries; migration order; current Copilot identity. Packages equal
the accepted parent byte for byte. No new live provider qualification is
claimed.
- **Fixtures and failures:** 34 fresh PostgreSQL 17 fixtures have 412
applied migration receipts through tag 0413_tired_kronos, plus fresh
per-call helper fixtures. All 52 original test hashes restore exactly.
The additional Photon test hash matches its passing proof. Earlier
remote chat passive-idle failure remains recorded; both passive/missing
cases pass in the new selection. Parent handoff timeout/fresh-file-pass
accounting remains in its audit. No assertion or timeout was weakened.
- [Integration and quiet producer
audit](https://github.com/paperclipai/paperclip/blob/305014b030d6c5eb891c70bc57036f4192d4fdad/doc/plans/2026-10-26-muse-terminal-settlement-sleep.md#current-master-integration-and-quiet-producers-2026-10-29).
Fresh CI, security, and exact-head Greptile 5/5 must pass before merge.

<details><summary>Earlier revision verification and failure
history</summary>

Receiver failure fairness: a controlled 1,100 ms batch reproduced a
30-second delay for healthy due receivers. The worker now supplies its
existing per-item deadline through `nextRunAtOnError`. All 20 receiver
lifecycle tests pass on the owner and combined tip
`31c12f1d68ff4a51e06fc2be2abf2c2aee9f9d3a`; server production
typechecking passes on the owner. The failed receiver retains its own
backoff, and settled receivers return to zero recurring SQL. The
five-second assertion budget is unchanged. See [the owning
audit](https://github.com/paperclipai/paperclip/blob/f7477933469fdc960f555425dce66eff25a94e54/doc/plans/2026-10-25-chat-receiver-discovery-sleep-audit.md#healthy-receiver-progress-after-a-slow-failed-batch).
Current-head CI and review remain required.

Approved delegated Test-tab calls now evaluate caller and target
policies in the existing claim transaction. Real HTTP regressions first
showed a missing caller charge and shared-limit self-rejection. All 78
connection-management cases and 92 gateway cases now pass on the owning
branch, along with server production typechecking. All 78
connection-management cases also pass at combined tip
`d25f830db89c1623bff3547bfc68015c1c66e0cf`. The three new cases cover
shared charges, separate charges, rollback on denial, duplicate
approval, and rejection of a later authorized call. Intermediate
counter/identity fixture failures and a reused-database Slack collision
are recorded in [the owning
audit](https://github.com/paperclipai/paperclip/blob/bbd31dea5ef3a4875950a87b4636af8b6bde858e/doc/idle-approved-tool-recovery.md#delegated-test-tab-admission-after-policy-changes).
The earlier combined checks below remain evidence for their stated
revisions. Current-head CI and review remain required.

Inherited voice-cleanup verification: the controlled slow-provider
regression and all 13 lifecycle tests passed on its owner and combined
tip `ab22d8ed0c09028315607414b83b375e5cc0a60a`. That repair changed only
the fixture clock advancement and its audit note.

Current head: `f7477933469fdc960f555425dce66eff25a94e54`. This remains a
single-duty PR stacked on #17772.

- Integrated pinned master `8548173fc9efdc0fa07644fb92dcc0b6851a95e5`
with normal merges. Every branch passed generated-schema checks and its
target-branch migration-order gate. Existing SQL is unchanged; the 25
stack migrations follow master’s new migration.
- Duty-specific producer coverage, lifecycle tests, and earlier failure
records:
[2026-10-24-recurring-server-sleep-stack.md](https://github.com/paperclipai/paperclip/blob/ff196791db6dada2dc04ea9db0d58c5c3beb4a41/doc/plans/2026-10-24-recurring-server-sleep-stack.md),
[2026-10-26-muse-terminal-settlement-sleep.md](https://github.com/paperclipai/paperclip/blob/ff196791db6dada2dc04ea9db0d58c5c3beb4a41/doc/plans/2026-10-26-muse-terminal-settlement-sleep.md).
- Combined tip `51e6cbe67979b55b340ea9f0f434c5b5c682ccd6`: **355
startup/cancellation/recovery/dispatch/preservation tests**, **75
connection-management tests**, and production server typechecking pass.
These combined checks do not mean every intermediate PR independently
ran the full repository suite.
- Real master-to-stack database upgrade passes: **379 unchanged prior
migration receipts plus 25 new migrations**, with the gateway
responsibility fixture preserved.
- Owning-branch checks: 84 runner installation/settlement cases plus
runner typechecking and generated TypeScript/Rust profile agreement; 253
GitHub review workflow cases. Stop-attribution and held-worker readiness
regressions failed before their fixes.
- Failure accounting: the initial combined run passed 349 cases and
failed six idle assertions in a reused preservation database containing
old cancelled runs. Repeating the combined command with a fresh
preservation database passed. Earlier invalid project selection, merge
import/type errors, and the interrupted full local suite are not counted
as passing checks.
- **Latest-head CI and review remain required.** Prior CI findings
include GitHub idle observation, route teardown, attention cleanup,
touch-picker E2E, workspace health, and voice-cleanup timing; local
passes do not erase those failures. No full local repository-suite,
deployed sleep, fleet cost, or live Copilot qualification claim is made.


</details>

## Risks

- In-process signals do not capture independent SQL or other replicas.
Supported external events still need ingress.
- Pending assignments and operation receipts retain their original
binding identity. A new agent configuration cannot take their ownership.
- Genuine qualification, OAuth, and recovery deadlines remain
obligations. They must not be discarded merely to make the worker quiet.
- The parent coarse-hint guard prevents recurring Muse discovery; real
company/binding authority changes retain their signals.
- No full local repository suite, full chat suite, live Muse or Copilot
qualification, deployed sleep, database suspension, external alarm
coverage, or cost savings is claimed. Existing Cloud blockers remain.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, code execution, real
PostgreSQL tests, and GitHub tools. The exact runtime model identifier
and context-window size are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Codex <noreply@openai.com>
 2026-10-29 06:18:40 -05:00
DottaPaperclipCodex
 		4d0847c4ca fix(server): settle external conversations from committed work (#17772) 
<!-- entire-trail-link-start -->
https://entire.com/gh/paperclipai/-/trails/6/changes/paperclip/6
<!-- entire-trail-link-end -->

## Thinking Path

> - Paperclip manages AI agents and the tasks that they execute.
> - External conversations must become idle after an authorized response
is published and all related work is complete.
> - Native replay and stranded-task recovery still discover this
settlement work through recurring scans.
> - A new deadline worker must preserve response ordering, monitor
handoffs, and unfinished work.
> - This pull request gives settlement one shared readiness predicate
and activates its worker from committed changes.
> - Existing rows provide recovery truth. No new table or per-feature
timer is added.
> - The benefit is a settlement worker that becomes quiet and resumes
when relevant work changes.

## Linked Issues or Issue Description

Refs #17758. This is slice 56. Slice 55 has merged. Related: #17606
(chat milestones), #17663 (bounded run reads), #17665 (monitor proof),
and #13809 (answered Slack conversation lifecycle). The provider
applicability remains Slack and Speko-owned conversations. This does not
extend Telegram settlement from #16559.

**What happened?**

Conversation settlement depended on periodic native replay and generic
task recovery. An exhausted monitor also cleared its schedule before
recovery admission. During that gap, an answered conversation could look
finished. Failed or deferred admission lost the monitor schedule.

**Expected behavior**

Relevant committed changes activate settlement. The worker uses the
owning service's evidence and becomes quiet when no settlement is owed.
Monitor recovery stays pending until admission, and restart does not
admit an already accepted recovery twice.

**Steps to reproduce**

1. Create an active external conversation with a successful run and an
authorized published response.
2. Keep execution, approval, a goal, or a monitor pending. Let the
settlement worker become quiet.
3. Resolve that prerequisite through its owning service. Verify that the
conversation settles without another periodic sweep.
4. Delay or fail exhausted-monitor recovery admission. Verify that the
monitor remains pending and the conversation cannot settle during the
handoff.

**Paperclip version or commit**

Accepted production source: `50f4432ed88537b8d522cf1dce24c33588db0bac`.
Parent #17758 merged as `8ac5a128e66fb672e88106412f76cf278599bfc4`. The
publication adopts that actual squash through a normal merge with an
unchanged complete child tree.

## What Changed

- Preserve quiet after coarse company-only maintenance hints. The shared
inbound-notice and response-availability helpers keep settlement enabled
by default. Only the milestone dispatcher suppresses their duplicate
settlement fanout. Its scoped issue-review signal and the explicit
company-authority signal still wake real work. Four database cases prove
both quiet and real company-wide producer delivery.
- Preserve source-generation checks for chat handoff,
task-before-comment locking, issue-scoped decision invalidation, and
current conversation event columns from master.
- Track initial external conversation settlement through shutdown
without blocking HTTP readiness on endpoint or task locks.

- Reuse the real queue's coalesced recovery receipt after restart,
without admitting the same monitor recovery again.

- Share the existing conversation readiness predicate between bounded
discovery and locked settlement. Preserve current provider policy,
run/publication proof, company boundaries, and endpoint-to-issue lock
order.
- Register a scoped deadline worker through the delivery coordinator.
Preserve startup, restore/worktree suppression, drain/resume, failure
recovery, and joined shutdown.
- Extend existing review, milestone, wake-notice, and finalization
signals. Add notifications at independent issue, comment, goal, session,
and wake-receipt unblocks.
- Keep monitor owner recovery pending until guarded admission. Reuse an
accepted receipt for the same monitor schedule after restart, even when
its limit reason changes; retain compatibility with legacy receipt keys.
Commit board escalation and its monitor clear together.
- Delegate periodic native settlement while preserving resource
callbacks. Keep direct/manual defaults and preserve generic recovery
ordering with a read-only readiness guard.
- Add the producer and lifecycle audit in
`doc/plans/2026-10-26-external-conversation-settlement-sleep.md` and
update the stack ledger.

## Verification

Accepted production source: `50f4432ed88537b8d522cf1dce24c33588db0bac`.
Publication head: `b256bb52647affd9f5279ea8c40f48bbe40f031f`.
The complete accepted child tree stays unchanged when adopting merged
parent #17758. This publication adds only the audit document.

- **3,129 distinct cases in 37 complete server files pass**, plus **18
selected chat-response cases** (3,147 selected cases). The other 1,796
chat cases were not run locally. Full settlement, queue, scheduling,
run-state, goals, interactions, task Retry, decision targets, status
reassessment, workspace, process recovery, and review consumers are
covered.
- **Preserved failure:** the first 35-file command passed 2,445 cases
and timed out one restored-work handoff at its existing 15-second limit.
That aggregate command failed. The unchanged full 89-case actions file
then passed on a fresh fixture; the affected case took 22 ms. Root cause
was not established. No timeout or assertion changed. The 34 other files
passed originally. Distinct counts exclude duplicate actions passes.
Fresh CI remains a separate gate.
- **Indirect wake regression:** the original unconditional baseline
failed two quiet checks. Guards alone left the chat case failing with
three candidate queries. The final controlled indirect baseline fails
one of four cases; the repair passes all four with the same test hash.
Positive controls preserve real company-only producers.
- **Source and fixture controls:** all 38 selected test files restore to
their exact committed hashes after fixture imports. Fresh PostgreSQL 17
fixtures use 413 migrations. All five manual merge resolutions preserve
current authority, locking, and event-column contracts.
- **Static checks pass:** database build; server typecheck and build;
module boundaries; migration order; Copilot profile identity. Runner
packages equal the accepted parent byte for byte. No new Runner/package
qualification is claimed.
- [Current integration and wake
audit](https://github.com/paperclipai/paperclip/blob/b256bb52647affd9f5279ea8c40f48bbe40f031f/doc/plans/2026-10-26-external-conversation-settlement-sleep.md#current-master-integration-and-indirect-wake-audit-2026-10-29).
Latest-head CI and exact-head Greptile 5/5 must pass before merge.

<details><summary>Earlier revision verification and preserved failure
history</summary>

Receiver failure fairness: a controlled 1,100 ms batch reproduced a
30-second delay for healthy due receivers. The worker now supplies its
existing per-item deadline through `nextRunAtOnError`. All 20 receiver
lifecycle tests pass on the owner and combined tip
`31c12f1d68ff4a51e06fc2be2abf2c2aee9f9d3a`; server production
typechecking passes on the owner. The failed receiver retains its own
backoff, and settled receivers return to zero recurring SQL. The
five-second assertion budget is unchanged. See [the owning
audit](https://github.com/paperclipai/paperclip/blob/83204a67880d980cfbea3771cee19a3cda0022b7/doc/plans/2026-10-25-chat-receiver-discovery-sleep-audit.md#healthy-receiver-progress-after-a-slow-failed-batch).
Current-head CI and review remain required.

Approved delegated Test-tab calls now evaluate caller and target
policies in the existing claim transaction. Real HTTP regressions first
showed a missing caller charge and shared-limit self-rejection. All 78
connection-management cases and 92 gateway cases now pass on the owning
branch, along with server production typechecking. All 78
connection-management cases also pass at combined tip
`d25f830db89c1623bff3547bfc68015c1c66e0cf`. The three new cases cover
shared charges, separate charges, rollback on denial, duplicate
approval, and rejection of a later authorized call. Intermediate
counter/identity fixture failures and a reused-database Slack collision
are recorded in [the owning
audit](https://github.com/paperclipai/paperclip/blob/ef9ae5767430d0093efc736ae0b303ba5afaae65/doc/idle-approved-tool-recovery.md#delegated-test-tab-admission-after-policy-changes).
The earlier combined checks below remain evidence for their stated
revisions. Current-head CI and review remain required.

Inherited voice-cleanup verification: the controlled slow-provider
regression and all 13 lifecycle tests passed on its owner and combined
tip `ab22d8ed0c09028315607414b83b375e5cc0a60a`. That repair changed only
the fixture clock advancement and its audit note.

Current head: `83204a67880d980cfbea3771cee19a3cda0022b7`. This remains a
single-duty PR stacked on #17758.

- Integrated pinned master `8548173fc9efdc0fa07644fb92dcc0b6851a95e5`
with normal merges. Every branch passed generated-schema checks and its
target-branch migration-order gate. Existing SQL is unchanged; the 25
stack migrations follow master’s new migration.
- Duty-specific producer coverage, lifecycle tests, and earlier failure
records:
[2026-10-24-recurring-server-sleep-stack.md](https://github.com/paperclipai/paperclip/blob/d6d8e4ffa6b330ebf4647aec997ee2dde6abc321/doc/plans/2026-10-24-recurring-server-sleep-stack.md),
[2026-10-26-external-conversation-settlement-sleep.md](https://github.com/paperclipai/paperclip/blob/d6d8e4ffa6b330ebf4647aec997ee2dde6abc321/doc/plans/2026-10-26-external-conversation-settlement-sleep.md).
- Combined tip `51e6cbe67979b55b340ea9f0f434c5b5c682ccd6`: **355
startup/cancellation/recovery/dispatch/preservation tests**, **75
connection-management tests**, and production server typechecking pass.
These combined checks do not mean every intermediate PR independently
ran the full repository suite.
- Real master-to-stack database upgrade passes: **379 unchanged prior
migration receipts plus 25 new migrations**, with the gateway
responsibility fixture preserved.
- Owning-branch checks: 84 runner installation/settlement cases plus
runner typechecking and generated TypeScript/Rust profile agreement; 253
GitHub review workflow cases. Stop-attribution and held-worker readiness
regressions failed before their fixes.
- Failure accounting: the initial combined run passed 349 cases and
failed six idle assertions in a reused preservation database containing
old cancelled runs. Repeating the combined command with a fresh
preservation database passed. Earlier invalid project selection, merge
import/type errors, and the interrupted full local suite are not counted
as passing checks.
- **Latest-head CI and review remain required.** Prior CI findings
include GitHub idle observation, route teardown, attention cleanup,
touch-picker E2E, workspace health, and voice-cleanup timing; local
passes do not erase those failures. No full local repository-suite,
deployed sleep, fleet cost, or live Copilot qualification claim is made.


</details>

## Risks

- Signals cover supported in-process writers. They do not supply change
capture for independent SQL or other replicas.
- Real company-wide changes can inspect active candidates. Coarse
maintenance hints do not activate settlement. Configured idle and
retained history must remain quiet.
- Monitor receipt binding and admission order must preserve company,
agent, issue, and exact schedule identity. Current parent handoff locks
and source-generation checks remain intact.
- The local restored-work timeout is retained above. A fresh full-file
pass does not establish its cause; latest-head CI must also pass.
- Generic recovery and Muse/resource duties remain separate slices. This
does not make the whole server idle. Existing Cloud sleep blockers
remain. No deployed sleep, database suspension, external alarm coverage,
fleet cost, or live provider qualification claim is made.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, code execution, PostgreSQL
tests, and GitHub tools. The exact runtime model identifier and
context-window size are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Codex <noreply@openai.com>
 2026-10-29 06:02:43 -05:00