PaperClipAI

Jannik/PaperClipAI

0 Fork 0

mirror of https://github.com/paperclipai/paperclip.git synced 2026-11-12 22:56:58 +01:00

Commit Graph

Author	SHA1	Message	Date
DottaandClaude Opus 5.5	f7c85bca18	test(runner-e2e): delete each runner E2E instance when it ends and expire kept instances after 6 hours (#20246 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart chaos harness boots a full temporary Paperclip instance per wave: embedded Postgres, workspaces and caches, often 5–13 GB > - At wave end it deleted that instance only when the wave passed with no harness error and no `--keep` > - Almost every chaos wave has at least one failing task, so almost every instance stayed on disk > - Over Nov 6–9, 359 `paperclip-runner-e2e-` directories filled the machine's disk (ENOSPC), which stopped a running storm wave > - This pull request deletes the instance at the end of every wave, smoke and baseline run unless it is explicitly kept or its processes could not be stopped, and expires kept instances after 6 hours > - The benefit is that chaos waves can run back to back without filling the disk ## Linked Issues or Issue Description What happened?* `main.ts` kept the temporary root whenever `keep \|\| anyFailure \|\| harnessError`. A failing task is the normal output of a chaos wave, so instances accumulated until the disk was full. The codex K13 storm w-20261109T113925-s9403 died after about a minute with every task cut off. Steps to reproduce 1. Run any wave in which at least one task fails. 2. The `paperclip-runner-e2e-` directory under the OS temp dir stays behind, with no expiry. Expected behavior* The instance is removed when the wave ends. Evidence is already in `results/`. Only an explicit `--keep`, or a stop failure that may leave processes running, keeps it, and kept instances expire. Actual behavior Every failing wave leaks its instance permanently. ## What Changed - New shared `tests/runner-e2e/instance-retention.ts`: - `releaseInstance` deletes the root unless `--keep` was passed or the processes could not be stopped. In those cases it writes a `.paperclip-runner-e2e-kept.json` marker (owner, reason, time) and sets the directory to 0700. An optional `retain` list keeps small evidence files and removes everything else. - `sweepExpiredKeptInstances` removes `paperclip-runner-e2e-*` directories that carry the marker, are older than 6 hours, and have no live `postmaster.pid` anywhere inside them, and that no live process command line references. Each root is made removable first, and one failed delete does not stop the sweep. Unmarked directories are never touched, so existing directories from older runs stay until someone decides on them. `recovery_state` markers never expire. - `createRunnerE2ETemporaryRoot` runs the sweep, so every caller gets it. - Callers: - A wave whose Daytona cleanup is still pending (environment DELETE blocked, sandboxes still running after cleanup, or cleanup errors) keeps its instance as `recovery_state`, because its database owns that cleanup. - The wave releases its instance before writing the ledger and summary, so a failed report write cannot skip cleanup. - Restart chaos `main.ts` (crash-point suite included) releases the instance in its final cleanup, which also runs on failure and interrupts. The crash-point task-count check moved above supervisor creation so that error no longer leaks a fresh root. - `ttft-baseline.ts` releases after stopping the supervisor, including on SIGINT and SIGTERM. - The rolling-deployment and controller-replacement smokes now delete their roots, keeping only the evidence report, logs and screenshots. If cleanup is unconfirmed, the root is kept with a marker. - `launch.ts` marks its unconfirmed-remote-cleanup root as `recovery_state`, which never expires, and its `PAPERCLIP_RUNNER_E2E_KEEP_FAILED_PRIVATE` root as `keep`. Its existing deletion path is unchanged, and the file gets one line shorter. - README: the instance lifecycle and `--keep`. ## Verification - `npx vitest run --config tests/runner-e2e/vitest.config.ts`: 328 files, 6915 passed. The new retention tests cover: - deleting a finished instance, and retaining only listed evidence; - keeping it with a marker and mode 0700; - sweeping only marked, expired instances whose Postgres isn't running, while sparing fresh, live, recovery-state, unmarked and unrelated directories; - a stale `postmaster.pid`; - sweeping on `createRunnerE2ETemporaryRoot`. The create-sweep test failed before `createRunnerE2ETemporaryRoot` ran the sweep. - `npx vitest run --config tests/restart-chaos/vitest.config.ts`: 221 passed. - `node scripts/typecheck-test-projects.mjs tests/runner-e2e` and `tests/restart-chaos`: clean. - No wave or smoke was run, because the disk is still nearly full. ## Risks - A failed wave's instance is no longer available for post-mortem DB inspection unless the wave ran with `--keep`. Evidence bundles in `results/` are unchanged. - The sweep relies on `postmaster.pid`. An instance whose Postgres was started outside the harness, without that file, could be removed after 6 hours. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code execution in Claude Code. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-09 09:20:09 -06:00
DottaandPaperclip	16787e4167	Qualify rolling web deployments and preserve Stop through handoff (#18526 ) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - The web server owns request handling and remote execution coordination. > - Deployment must start a ready replacement while the old server still serves. > - An outage test does not prove that two live servers can overlap safely. > - This pull request adds a separate Product E2E cell for that boundary. > - The existing run, file, recovery, and retained-session checks still apply. ## Linked Issues or Issue Description Refs #18518. The runtime prerequisite is merged; this PR qualifies its ephemeral server and deployment contracts. ## What Changed - Add the explicit `rolling-web-deployment` Codex Daytona workflow. - Start independent A/B/C homes and compiled builds behind a test ingress. - Wait for readiness before routing, then drain and delete the old home. - Observe active authority before handoff and use the existing read-only recovery oracle. - Require ordinary reads and saves to succeed throughout replacement. Read back every accepted save exactly once. - Preserve the five-minute outage suites and their existing validators. - Add a model-free supervisor smoke and an unready replacement control. - Require graceful exit before deleting an old home; forced cleanup cannot satisfy the oracle. - Preserve a durable Stop through lease release/expiry until the recovered execution owner acknowledges it. Permit only fenced reattachment of the original execution for Stop cleanup; submit no new turn. - Exclude events saved by A during shutdown from evidence of B making progress. ## Verification - E2E TypeScript check passed. - Affected harness checks: 143 passed. Catalog, fixture and report checks: 120 passed. - Actual supervisor smoke passed two replacements: 436 reads, 45 saves, zero request failures, and confirmed owned resource cleanup. - B became ready in 18.7 seconds; C became ready in 109.2 seconds. A prior C timeout remains failed evidence. - Fresh browser proof: 11 checks passed, 246 reads, 50 acknowledged saves, zero failures, 1,058ms reconnect, both old homes deleted and cleanup confirmed. - Stop lease-gap regression: three PostgreSQL delivery tests passed with a simulated provider edge. Current server TypeScript build/vendoring passed. - Full harness: 5,814 passed, 17 failed, four skipped. Two shutdown races passed the focused file; two report timeouts passed an isolated 42-test rerun. Onboarding/workflow-security/Pi/date-expired fixtures are unchanged on inspected master; the host embedded PostgreSQL fixture remains a setup limitation. This is not a green full local suite. - CI at `172509b8be` passed all gates. Greptile is 5/5 with every review thread resolved. The earlier access-service race failure passed its isolated regression and the subsequent full CI. - Final review corrections: six focused executor cases, three PostgreSQL command-delivery cases, 19 harness cases and server/E2E typechecks passed. - Three live attempts failed before agent creation: two harness setup gaps were fixed, then provider authentication failed. The saved OpenAI key returns 401; a refresh is pending. The Daytona probe reported cleanup uncertain, and no sandbox with that attempt label was found through the authenticated staging manager. No live model execution is claimed. - Live Runner overlap, released-container qualification and hosted deployment proof remain pending. Merging this opt-in qualification lane does not claim those results. ## Risks - The source lane shares workspace dependencies and does not establish container packaging correctness. - The live workflow needs an explicitly owned S3 bucket, PostgreSQL, pinned remote image and provider credentials. - Qualification cleanup stops all owned processes after failure. A failed replacement must not be reported as a pass. ## Model Used OpenAI Codex, GPT-6. Tool use, code execution and repository review. The runtime does not expose a more specific model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-11-01 02:53:27 -06:00
DottaandPaperclip	6987c2f20b	test(upgrades): qualify controller replacement with immutable artifacts and phase evidence (#17455 ) <!-- entire-trail-link-start --> https://entire.com/gh/paperclipai/-/trails/105/changes/paperclip/105 <!-- entire-trail-link-end --> ## Thinking Path > - Paperclip needs evidence that an admitted execution survives a main-server replacement. > - Runner mechanics and server adoption belong to #17097 and #17439. > - A healthy HTTP endpoint does not prove recovery of the original execution. > - Qualification must identify the serving artifact, natural ownership, authenticated attachment and resumed processing separately. > - It must also retain failed runs and measure normal-operation costs. > - This PR extends the existing qualification machinery and acceptance record. ## Linked Issues or Issue Description What existing behavior does this improve? Operators can assess upgrade continuity from phase evidence tied to exact artifacts and profiles. Current behavior A same-artifact restart or synthetic executor pass cannot establish different-artifact upgrade and rollback support. Idle SQL gates also do not measure active-run capacity. Proposed behavior Use immutable A/B artifacts and strict phase receipts through the existing supervisor. Preserve failed evidence and add measured before/after workload costs. Builds on merged #17097 and #17439. The base is now `master`. #17456 follows for review convenience. This PR lands qualification machinery and the evidence record. The remaining hosted gates stay open for deployment qualification; merging this machinery does not claim those experiments passed. ## What Changed - Extend the main-server supervisor and installed-artifact verification for actual serving revisions, forward migrations and compatible rollback. - Distinguish HTTP readiness, natural ownership, authenticated attachment, resumed processing and backlog settlement. - Retain the original reconnect grace and execution identity checks. Never substitute a new task or edited lease for continuity. - Reuse real-harness and Daytona qualification paths, with exact profile and artifact provenance. - Bind current Copilot attestation inputs in the Daytona image fingerprint. - Keep one acceptance record, including both failed file-only stress phases and the final passing bounded-pool comparison. ## Verification After retargeting to master, all 158 focused qualification cases across 13 files pass, with E2E typecheck. The first 12-file batch has four report-lock test failures and 133 passes. Master #17811 had committed the exact already-approved output lock, while that fixture assumed HEAD must be the uncorrected input. The fixture now checks both pinned forms, executes the committed-lock path, and reconstructs the pinned input to retain all rejection tests. The production helper, policy hashes, patch and dependency resolution are unchanged. The corrected batch explicitly includes both process-preservation suites; the prior 146-case receipt has its own source and selection. Current candidate `8f1d6ec02f` integrates master `09ad9adce2`, including the merged server slice. Its diff contains only qualification code and documentation. The [qualification record](https://github.com/paperclipai/paperclip/blob/8f1d6ec02f5bf6c90e7b27da8156953cabd9a180/doc/plans/2026-10-22-hot-upgrade-qualification.md) is the acceptance ledger and retains source-attributed failures and hosted limits. On the eighth integration, all 146 cases across 12 focused qualification files pass, including report-lock preparation, workflow security and workflow targeting. E2E typecheck passes. The earlier 118-case result remains attributed to its prior source. The first sandbox invocation has 117 passes and one failure because process inspection was denied during inert fixture cleanup. The permitted repeat passes all 118 and a targeted process check finds no remaining fixture daemon. The earlier 96-case receipt remains source-attributed in the ledger. The latest two review regressions first fail: an intermediate replacement can cite another runner/session despite a valid final step, and malformed receipt JSON aborts evidence packaging. Each replacement now binds run/company/issue/runner/session, and corrupt receipts retain explicit invalid metadata plus sanitized diagnostics while other logs and receipts still package. These checks prove qualification machinery with injected supervisor operations, configuration subprocesses and real filesystem packaging. They do not qualify a hosted harness. Three request-reader cases, 14 artifact/configuration/cache/history cases, and three lost-partial-receipt cases fail before their respective corrections. The final collector preserves sanitized startup, readiness and recovery phase receipts through packaging and temporary-root deletion. Earlier embedded-Postgres initialization, report-catalog drift, CI scheduling/cleanup failures and fixture collisions remain recorded with their corrections or limits. The previous Greptile service error is retained as historical review evidence, not a current review pass. The record includes prior source `9bb9d3417e`'s actual Linux contention/fairness successes and the ordinary-warm regression proof from the server slice. The historical 300-session component campaign passes four after phases and retains earlier burst failures. Those component costs do not establish hosted capacity. The hosted API-only pair has zero agents/viewers, fails both absolute latency gates and the candidate memory gate, and does not complete the active-agent/viewer matrix. The strict ten-agent same-artifact restart remains attributed to its original artifact. It does not qualify this updated stack. Actual A→B/B→A, the normal drain-free Cloud rollout, and other hosted harnesses remain open. No new provider turn, deployment or hosted qualification was run during merge preparation. Prior source `57d3767eac` jobs 122590643944, 122590643983 and 122590644161 ended after runner shutdown/cancellation without final Vitest results. The shutdown cause is unproven. No retry was issued before this source was superseded by the required server recovery fix. These attempts remain failures/incomplete evidence. Fresh exact-head CI, legacy status, completed Greptile 5/5, resolved feedback and normal merge eligibility remain required. ## Risks Different-artifact A→B/B→A, the normal drain-free Cloud rollout path, other hosted harnesses and the complete concurrency/viewer matrix remain open. Frozen branch builds are development-build evidence. This PR owns qualification and evidence; production defects must be fixed in their earlier owning slice. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools and code execution. The runtime does not expose a more specific model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-27 21:58:09 -05:00

Author

SHA1

Message

Date

DottaandClaude Opus 5.5

f7c85bca18

test(runner-e2e): delete each runner E2E instance when it ends and expire kept instances after 6 hours (#20246 )

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart chaos harness boots a full temporary Paperclip instance
per wave: embedded Postgres, workspaces and caches, often 5–13 GB
> - At wave end it deleted that instance only when the wave passed with
no harness error and no `--keep`
> - Almost every chaos wave has at least one failing task, so almost
every instance stayed on disk
> - Over Nov 6–9, 359 `paperclip-runner-e2e-*` directories filled the
machine's disk (ENOSPC), which stopped a running storm wave
> - This pull request deletes the instance at the end of every wave,
smoke and baseline run unless it is explicitly kept or its processes
could not be stopped, and expires kept instances after 6 hours
> - The benefit is that chaos waves can run back to back without filling
the disk

## Linked Issues or Issue Description

**What happened?**
`main.ts` kept the temporary root whenever `keep || anyFailure ||
harnessError`. A failing task is the normal output of a chaos wave, so
instances accumulated until the disk was full. The codex K13 storm
w-20261109T113925-s9403 died after about a minute with every task cut
off.

**Steps to reproduce**
1. Run any wave in which at least one task fails.
2. The `paperclip-runner-e2e-*` directory under the OS temp dir stays
behind, with no expiry.

**Expected behavior**
The instance is removed when the wave ends. Evidence is already in
`results/`. Only an explicit `--keep`, or a stop failure that may leave
processes running, keeps it, and kept instances expire.

**Actual behavior**
Every failing wave leaks its instance permanently.

## What Changed

- New shared `tests/runner-e2e/instance-retention.ts`:
- `releaseInstance` deletes the root unless `--keep` was passed or the
processes could not be stopped. In those cases it writes a
`.paperclip-runner-e2e-kept.json` marker (owner, reason, time) and sets
the directory to 0700. An optional `retain` list keeps small evidence
files and removes everything else.
- `sweepExpiredKeptInstances` removes `paperclip-runner-e2e-*`
directories that carry the marker, are older than 6 hours, and have no
live `postmaster.pid` anywhere inside them, and that no live process
command line references. Each root is made removable first, and one
failed delete does not stop the sweep. Unmarked directories are never
touched, so existing directories from older runs stay until someone
decides on them. `recovery_state` markers never expire.
- `createRunnerE2ETemporaryRoot` runs the sweep, so every caller gets
it.
- Callers:
- A wave whose Daytona cleanup is still pending (environment DELETE
blocked, sandboxes still running after cleanup, or cleanup errors) keeps
its instance as `recovery_state`, because its database owns that
cleanup.
- The wave releases its instance before writing the ledger and summary,
so a failed report write cannot skip cleanup.
- Restart chaos `main.ts` (crash-point suite included) releases the
instance in its final cleanup, which also runs on failure and
interrupts. The crash-point task-count check moved above supervisor
creation so that error no longer leaks a fresh root.
- `ttft-baseline.ts` releases after stopping the supervisor, including
on SIGINT and SIGTERM.
- The rolling-deployment and controller-replacement smokes now delete
their roots, keeping only the evidence report, logs and screenshots. If
cleanup is unconfirmed, the root is kept with a marker.
- `launch.ts` marks its unconfirmed-remote-cleanup root as
`recovery_state`, which never expires, and its
`PAPERCLIP_RUNNER_E2E_KEEP_FAILED_PRIVATE` root as `keep`. Its existing
deletion path is unchanged, and the file gets one line shorter.
- README: the instance lifecycle and `--keep`.

## Verification

- `npx vitest run --config tests/runner-e2e/vitest.config.ts`: 328
files, 6915 passed. The new retention tests cover:
  - deleting a finished instance, and retaining only listed evidence;
  - keeping it with a marker and mode 0700;
- sweeping only marked, expired instances whose Postgres isn't running,
while sparing fresh, live, recovery-state, unmarked and unrelated
directories;
  - a stale `postmaster.pid`;
  - sweeping on `createRunnerE2ETemporaryRoot`.

The create-sweep test failed before `createRunnerE2ETemporaryRoot` ran
the sweep.
- `npx vitest run --config tests/restart-chaos/vitest.config.ts`: 221
passed.
- `node scripts/typecheck-test-projects.mjs tests/runner-e2e` and
`tests/restart-chaos`: clean.
- No wave or smoke was run, because the disk is still nearly full.

## Risks

- A failed wave's instance is no longer available for post-mortem DB
inspection unless the wave ran with `--keep`. Evidence bundles in
`results/` are unchanged.
- The sweep relies on `postmaster.pid`. An instance whose Postgres was
started outside the harness, without that file, could be removed after 6
hours.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

2026-11-09 09:20:09 -06:00

DottaandPaperclip

16787e4167

Qualify rolling web deployments and preserve Stop through handoff (#18526 )

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The web server owns request handling and remote execution
coordination.
> - Deployment must start a ready replacement while the old server still
serves.
> - An outage test does not prove that two live servers can overlap
safely.
> - This pull request adds a separate Product E2E cell for that
boundary.
> - The existing run, file, recovery, and retained-session checks still
apply.

## Linked Issues or Issue Description

Refs #18518. The runtime prerequisite is merged; this PR qualifies its
ephemeral server and deployment contracts.

## What Changed

- Add the explicit `rolling-web-deployment` Codex Daytona workflow.
- Start independent A/B/C homes and compiled builds behind a test
ingress.
- Wait for readiness before routing, then drain and delete the old home.
- Observe active authority before handoff and use the existing read-only
recovery oracle.
- Require ordinary reads and saves to succeed throughout replacement.
Read back every accepted save exactly once.
- Preserve the five-minute outage suites and their existing validators.
- Add a model-free supervisor smoke and an unready replacement control.
- Require graceful exit before deleting an old home; forced cleanup
cannot satisfy the oracle.
- Preserve a durable Stop through lease release/expiry until the
recovered execution owner acknowledges it. Permit only fenced
reattachment of the original execution for Stop cleanup; submit no new
turn.
- Exclude events saved by A during shutdown from evidence of B making
progress.

## Verification

- E2E TypeScript check passed.
- Affected harness checks: 143 passed. Catalog, fixture and report
checks: 120 passed.
- Actual supervisor smoke passed two replacements: 436 reads, 45 saves,
zero request failures, and confirmed owned resource cleanup.
- B became ready in 18.7 seconds; C became ready in 109.2 seconds. A
prior C timeout remains failed evidence.
- Fresh browser proof: 11 checks passed, 246 reads, 50 acknowledged
saves, zero failures, 1,058ms reconnect, both old homes deleted and
cleanup confirmed.
- Stop lease-gap regression: three PostgreSQL delivery tests passed with
a simulated provider edge. Current server TypeScript build/vendoring
passed.
- Full harness: 5,814 passed, 17 failed, four skipped. Two shutdown
races passed the focused file; two report timeouts passed an isolated
42-test rerun. Onboarding/workflow-security/Pi/date-expired fixtures are
unchanged on inspected master; the host embedded PostgreSQL fixture
remains a setup limitation. This is not a green full local suite.
- CI at `172509b8be` passed all gates. Greptile is 5/5 with every review
thread resolved. The earlier access-service race failure passed its
isolated regression and the subsequent full CI.
- Final review corrections: six focused executor cases, three PostgreSQL
command-delivery cases, 19 harness cases and server/E2E typechecks
passed.
- Three live attempts failed before agent creation: two harness setup
gaps were fixed, then provider authentication failed. The saved OpenAI
key returns 401; a refresh is pending. The Daytona probe reported
cleanup uncertain, and no sandbox with that attempt label was found
through the authenticated staging manager. No live model execution is
claimed.
- Live Runner overlap, released-container qualification and hosted
deployment proof remain pending. Merging this opt-in qualification lane
does not claim those results.

## Risks

- The source lane shares workspace dependencies and does not establish
container packaging correctness.
- The live workflow needs an explicitly owned S3 bucket, PostgreSQL,
pinned remote image and provider credentials.
- Qualification cleanup stops all owned processes after failure. A
failed replacement must not be reported as a pass.

## Model Used

OpenAI Codex, GPT-6. Tool use, code execution and repository review. The
runtime does not expose a more specific model ID or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

2026-11-01 02:53:27 -06:00

DottaandPaperclip

6987c2f20b

test(upgrades): qualify controller replacement with immutable artifacts and phase evidence (#17455 )

https://entire.com/gh/paperclipai/-/trails/105/changes/paperclip/105

## Thinking Path

> - Paperclip needs evidence that an admitted execution survives a
main-server replacement.
> - Runner mechanics and server adoption belong to #17097 and #17439.
> - A healthy HTTP endpoint does not prove recovery of the original
execution.
> - Qualification must identify the serving artifact, natural ownership,
authenticated attachment and resumed processing separately.
> - It must also retain failed runs and measure normal-operation costs.
> - This PR extends the existing qualification machinery and acceptance
record.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Operators can assess upgrade continuity from phase evidence tied to
exact artifacts and profiles.

**Current behavior**

A same-artifact restart or synthetic executor pass cannot establish
different-artifact upgrade and rollback support. Idle SQL gates also do
not measure active-run capacity.

**Proposed behavior**

Use immutable A/B artifacts and strict phase receipts through the
existing supervisor. Preserve failed evidence and add measured
before/after workload costs.

Builds on merged #17097 and #17439. The base is now `master`. #17456
follows for review convenience. This PR lands qualification machinery
and the evidence record. The remaining hosted gates stay open for
deployment qualification; merging this machinery does not claim those
experiments passed.

## What Changed

- Extend the main-server supervisor and installed-artifact verification
for actual serving revisions, forward migrations and compatible
rollback.
- Distinguish HTTP readiness, natural ownership, authenticated
attachment, resumed processing and backlog settlement.
- Retain the original reconnect grace and execution identity checks.
Never substitute a new task or edited lease for continuity.
- Reuse real-harness and Daytona qualification paths, with exact profile
and artifact provenance.
- Bind current Copilot attestation inputs in the Daytona image
fingerprint.
- Keep one acceptance record, including both failed file-only stress
phases and the final passing bounded-pool comparison.

## Verification

After retargeting to master, all 158 focused qualification cases across
13 files pass, with E2E typecheck. The first 12-file batch has four
report-lock test failures and 133 passes. Master #17811 had committed
the exact already-approved output lock, while that fixture assumed HEAD
must be the uncorrected input. The fixture now checks both pinned forms,
executes the committed-lock path, and reconstructs the pinned input to
retain all rejection tests. The production helper, policy hashes, patch
and dependency resolution are unchanged. The corrected batch explicitly
includes both process-preservation suites; the prior 146-case receipt
has its own source and selection.

Current candidate `8f1d6ec02f` integrates master `09ad9adce2`, including
the merged server slice. Its diff contains only qualification code and
documentation. The [qualification
record](https://github.com/paperclipai/paperclip/blob/8f1d6ec02f5bf6c90e7b27da8156953cabd9a180/doc/plans/2026-10-22-hot-upgrade-qualification.md)
is the acceptance ledger and retains source-attributed failures and
hosted limits.

On the eighth integration, all 146 cases across 12 focused qualification
files pass, including report-lock preparation, workflow security and
workflow targeting. E2E typecheck passes. The earlier 118-case result
remains attributed to its prior source. The first sandbox invocation has
117 passes and one failure because process inspection was denied during
inert fixture cleanup. The permitted repeat passes all 118 and a
targeted process check finds no remaining fixture daemon. The earlier
96-case receipt remains source-attributed in the ledger. The latest two
review regressions first fail: an intermediate replacement can cite
another runner/session despite a valid final step, and malformed receipt
JSON aborts evidence packaging. Each replacement now binds
run/company/issue/runner/session, and corrupt receipts retain explicit
invalid metadata plus sanitized diagnostics while other logs and
receipts still package. These checks prove qualification machinery with
injected supervisor operations, configuration subprocesses and real
filesystem packaging. They do not qualify a hosted harness.

Three request-reader cases, 14 artifact/configuration/cache/history
cases, and three lost-partial-receipt cases fail before their respective
corrections. The final collector preserves sanitized startup, readiness
and recovery phase receipts through packaging and temporary-root
deletion. Earlier embedded-Postgres initialization, report-catalog
drift, CI scheduling/cleanup failures and fixture collisions remain
recorded with their corrections or limits. The previous Greptile service
error is retained as historical review evidence, not a current review
pass.

The record includes prior source `9bb9d3417e`'s actual Linux
contention/fairness successes and the ordinary-warm regression proof
from the server slice. The historical 300-session component campaign
passes four after phases and retains earlier burst failures. Those
component costs do not establish hosted capacity. The hosted API-only
pair has zero agents/viewers, fails both absolute latency gates and the
candidate memory gate, and does not complete the active-agent/viewer
matrix.

The strict ten-agent same-artifact restart remains attributed to its
original artifact. It does not qualify this updated stack. Actual
A→B/B→A, the normal drain-free Cloud rollout, and other hosted harnesses
remain open. No new provider turn, deployment or hosted qualification
was run during merge preparation.

Prior source `57d3767eac` jobs 122590643944, 122590643983 and
122590644161 ended after runner shutdown/cancellation without final
Vitest results. The shutdown cause is unproven. No retry was issued
before this source was superseded by the required server recovery fix.
These attempts remain failures/incomplete evidence. Fresh exact-head CI,
legacy status, completed Greptile 5/5, resolved feedback and normal
merge eligibility remain required.

## Risks

Different-artifact A→B/B→A, the normal drain-free Cloud rollout path,
other hosted harnesses and the complete concurrency/viewer matrix remain
open. Frozen branch builds are development-build evidence. This PR owns
qualification and evidence; production defects must be fixed in their
earlier owning slice.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository tools and code
execution. The runtime does not expose a more specific model ID or
context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [ ] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

2026-10-27 21:58:09 -05:00