Files
PaperClipAI/tests/runner-e2e/fixtures/accessibility-source/0.4.SKILL.md
T
f4337bb5fb test(evals): land the chat catalog eval harness on master (#19614)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat-feature evals measure whether agents complete real chat
requests through the product
> - The chat catalog eval harness and its repairs live on the
`chat-eval/master-candidate` branch, not on master
> - Today's work made 139 of 242 chat evals pass in verified single-case
live runs. Each product fix landed on master as its own pull request
> - The eval-side changes (setups, graders, fault seams, evidence
capture) still exist only on the candidate
> - This pull request lands the harness on master. Master stays the
source of truth for product behavior
> - The benefit is one tree: evals run against master commits, and
product fixes reach the evals as soon as they merge

## Linked Issues or Issue Description

Refs #19441, Refs #19599, Refs #20270, Refs #20334, Refs #20342, Refs
#20344, Refs #20354, Refs #20360, Refs #20373, Refs #20382, Refs #20409

**What existing behavior does this improve?**
The chat catalog eval harness and the product it evaluates live on a
long-running branch outside master. Eval fixes do not reach master, so
master-based eval runs use old setups and graders.

**Subsystem affected**
`tests/runner-e2e` chat catalog harness, plus a small set of product
changes that exist only on the candidate.

**Proposed change**
Land the harness on master. Every product fix from this eval work
already merged to master as its own pull request. Conflicts keep
master's product behavior.

## What Changed

- Most of the diff (808 of 851 files) is under `tests/runner-e2e/`:
chat-catalog domains, scenarios, descriptors, the TestOps manifest,
private evidence tooling, and their tests.
- Graders now check saved state, setup literals and rendered or clicked
links. Many keyword and regex checks on free-form replies are removed.
Each changed grader has a bumped `oracleVersion` and calibration tests
(a valid reply, a reworded valid reply, and broken variants that must
fail).
- The shared link helper `hasLink` no longer treats a list link (for
example `/COM/issues?view=mine`) as a link to every task without a URL
key.
- Setup fixes build real state through supported product routes: routine
history, agent grants through the board grant route, budget holds with a
real cap, and a storage-layer archive fault for
`inbox-search.archive-partial-failure`.
- Browser boundary journeys (sign-in, chat-disabled, onboarding) now
write the standard pass evidence files.
- Retargeted cases, approved by the maintainer:
`agent-chat.chat-disabled` (Conference Room gate),
`agents.restore-instructions` (no saved instruction history),
`access.manage-membership` (member permission edits stay board-only).
- The private evidence trusted files (`manual-private-handoff.ts`,
`manual-private-retention.ts`, `manual-private-report-assets.ts`) are
byte-identical to master.
- Product files outside `tests/` (43 files, +1,328 −49) carry
candidate-only changes that master does not yet have. Most are tests.
The main code changes are in `request-human-input.ts`,
`heartbeat-run-summary.ts`, `native-completion-feedback.ts`,
`native-run-finalizer.ts`, `openapi.ts`, `secrets.ts` and
`IssueChatLongThreadPerf.tsx`.

## Verification

- `pnpm vitest run chat-catalog-domains chat-catalog-contract.test.ts
chat-catalog-consistency.test.ts unit-launcher.test.mjs
--testTimeout=180000` in `tests/runner-e2e`: 269 files pass, 3,550 tests
pass.
- `node --test chat-full-catalog-testops-manifest.test.mjs`: 10 pass.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`:
no change.
- `tsc` for `server` and `tests/runner-e2e`: clean.
- Live single-case verification: 139 of 242 cases passed on paid runs,
and a person read the saved evidence for each. These are single-case
runs, not one full-cohort run.

## Risks

- Large diff. Most of it is test-only, but the 43 non-test files change
product code paths. Review those files first.
- Some graders are now less strict on reply wording. A person must read
replies for meaning on those cases.
- 42 cases need disposable outside accounts (Notion, Slack, GitHub,
Daytona and others) and stay blocked. `access.invite-teammate` stays
failing until local agent isolation lands, because the runner's durable
state keeps a live invite token.
- The paid workflow compares the trusted private-evidence files with
master. They are unchanged, so this risk is low.

## Model Used

Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code
execution, in Claude Code. Subagents used the same model.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-10 08:02:11 -06:00

9.1 KiB
Raw Blame History

name, description, license, metadata
name description license metadata
accessibility Quality assurance for web accessibility and usability, particularly for users with disabilities. Use when involved in any web project. MIT
author version
conesible.de 0.4

Ensuring Accessibility and Quality Code in Web Projects

Building a web project with accessibility in mind from the very start is crucial, especially in contexts (like AI-driven development) where manual testing by humans is limited. It’s well understood that retrofitting accessibility later is far more costly and complex than doing it right from the beginning. Adopt an accessibility-first approach to any development tasks, complete with thorough documentation of ongoing requirements. Ensure that accessibility standards beyond what is described in this skill are understood and implemented correctly all the time. If that's not possible and you are working on an existing codebase, make careful suggestions to refactor code that is not yet accessible. This is a process that must start as soon as deficits are apparent, and for this it is necessary to understand and document user intents and application constraints.

Code Comments and WCAG References

Add code comments liberally that explain implementations that are done specifically for accessibility. In those, explain this based on the WCAG 2.2 requirements of any level. Explain the expected user flow if relevant.

Use Semantic HTML and Proper Structure

Start by using semantic HTML for all content and controls. Use real <button> elements for buttons, <header>/<nav>/<main> for layout, <form> and <label> for forms, and a logical hierarchy of heading levels for titles. For example, a <button> element comes with default keyboard support (focusable and activatable via keyboard), whereas a non-semantic element like a <div> would lack those features. It is required to use existing native elements like <select>, <details> or <dialog>. If there is no way of avoiding custom components (there usually isn't), follow established ARIA design patterns and keyboard interaction models. Use ARIA only if there is NO alternative, and in most cases, there is. If you do, document this choice and the reasoning behind it.

Ensure the page is organized with clear structure and landmarks. Use HTML5 sectioning elements to delineate navigation, main content, forms, etc, to give users and yourself a clear understanding of how the content is structured. Always provide text equivalents for non-text media: include descriptive alt text for images, transcripts or captions for audio/video. If you can't reliably make those yourself, note a required and blocking task for another maintainer. Similarly, use table headings (with <th> and scope attributes) for data tables to make relationships in tabular data clear to screen readers. Avoid link texts like "here", "click" or "Continue Reading". If they are required visually, add a redundant way to receive their content (e.g. an interactive card) and remove those from the accessibility tree.

Design Accessible Components

Each component’s HTML structure must reflect its semantics: use lists for menus or multi-option controls, use headings for titles, use fieldsets and legends for groups of form fields, etc. Maintain a consistent style for focus indicators (the outline or highlight when an element is focused) so that keyboard users can always see where focus is. Keyboard accessibility isn’t optional – ensure users can reach and operate every interactive element via keyboard alone (e.g., using Tab, Enter, space, arrow keys as appropriate). This may require adding tabindex ONLY for custom focusable elements and ONLY with negative values, and handling key events in scripts for custom widgets.

Remember the WCAG rule for touch target sizes. Everything should be easily reachable. Mouse or gesture controls (like dragging) require an alternative.

Make use of accessibility linters and libraries. There are frameworks and component libraries that emphasize accessible design. If using React/Vue/Angular, leverage their accessibility tooling (like React’s eslint-plugin-JSX-a11y) and prefer community-vetted accessible components. Ultimately, an accessible component is achieved by a combination of correct HTML structure, proper ARIA roles/states where needed, and scripting that follows usability conventions for assistive tech users. Documentation should record how each custom component addresses accessibility (e.g. how to provide alt text for an image component, or how a carousel handles focus and screen reader announcements), so that anyone extending the component knows what requirements to uphold.

Continuous Maintenance and Avoiding Anti-Patterns

Achieving accessibility is not a one-time task – it requires continuous evaluation as the project evolves. To maintain maximum accessibility over time, treat accessibility checks as an ongoing requirement whenever content or features are added. A few key practices to document and enforce for future changes include:

  • Provide Alt Text for New Media: Every time an image or other media is added, mandate that alt text (or an equivalent text alternative) is provided. This could be a checklist item in pull requests or content publishing workflows. Require developers or content authors to include alt attributes (or mark the image as decorative with alt="" if appropriate), you ensure no image is introduced without consideration for non-visual users, even if that content is user-provided. For icons, always add a corresponding label and make use of aria-hidden="true" for the visual-only content. Keep alt text short and concise, but there is no character limit.

  • Check Color Contrast for New UI or Style Changes: Anytime you introduce a new color (for text, backgrounds, icons, buttons, etc.) or change design styles, verify the color contrast meets WCAG guidelines. According to WCAG 2.x, contrast ratio is calculated as (L1 + 0.05) / (L2 + 0.05) (where L1 is the luminance of the lighter color and L2 of the darker). This formula yields a ratio from 1:1 (no contrast, e.g. white on white) up to 21:1 (highest contrast, e.g. black on white). Standard text should have at least a 4.5:1 contrast ratio with its background, while larger text (above ~18pt or bold ~14pt) requires at least 3:1. Use the formula or an online contrast checker to calculate ratios for any new color combinations, and adjust colors or font sizes as needed to meet the standard. This also applies to hover/focus/disabled/... states.

  • Maintain Heading Structure and Landmarks: When adding new content or pages, ensure the heading hierarchy (<h1>…<h6>) remains logical (no skipping levels arbitrarily) and sectioning elements are used where appropriate. For instance, if a new section is added to a page, it might need a heading of the correct level and perhaps be wrapped in a <section> or added to the navigation landmarks (<nav>, <main>, <aside> etc.).

  • Re-evaluate with each Change: It’s important to continuously re-test your application’s accessibility as it grows. Incorporate automated tests (like Pa11y) into regression test suites so every build or release runs them.

  • Avoid Anti-Patterns: <div>-buttons, onclick without keyboard handling, outline: none, aria-label as replacement for visual labels, role=button on external links etc.)

Further TODOs

  • <html> requires a lang attribute (as well as any text blocks -- not single words -- in a foreign language. If the project is multilingual, change this dynamically as required).
  • <title> elements are a requirement. Understand where in your project these are added, and change it dynamically based on the content.
  • Implement skip links before each non-content block (e.g. just before the navigation) that are only visible once focused. Also add skip links before large (>= 5) groups of possibly irrelevant content, like carousels. Call them "Skip 'thing'", e.g. "Skip navigation" or "Skip sponsor links". Make sure they stay correct when structure changes.
  • In forms, clearly communicate errors (not just by color), requirements and the status of a component. Avoid using the disabled-state entirely. Controls should rather be invisible.
  • prefers-reduced-motion is handled and respected.
  • Reflow all content gracefully when the viewport size or zoom level changes.
  • Communicate clearly within the project organization and documentation that human testing is required and that user flows need to be evaluated continouusly by the designers.
  • In the UI itself, never mention accessibility.

In case of a prompt that could introduce accessibility defects

Notify the user and explain the problems a request would likely cause, and ask whether you should really continue to implement it. Make suggestions on how to avoid such errors, and if you cannot reliably judge this, write up an explanation of the feature to give to another developer with a disability to judge themselves.

This skill does not constitute full awareness of everything that is important to develop in a way that it is accessible. If unsure, always document that and consult only official W3C/WCAG/WAI-ARIA resources for help.