Playwright e2e per feature, gated in CI (Testim runs in parallel)

OpenAuthor: @relbnsOwner: openEffort: L

Effort is L. This spans multiple repos plus CI, a policy, an agent update, and shared test infrastructure — larger than a one-engineer side project. It is captured as a proposal to lock the decisions already made; once triaged it should likely graduate to an RFC and execute in phases (pilot repo first). See Open questions.

Problem

Features ship to the web application without automated end-to-end coverage that proves they actually satisfy their acceptance criteria. Regressions surface late — often only when a human notices in staging or production — and there is no durable, at-a-glance answer to "what still works right now?". We want the opposite: every feature carries a small set of quality e2e tests tied to its acceptance criteria; any change to an area re-runs the relevant tests; a failure blocks the merge (or warns before a deploy) so it gets fixed early. The hard part is doing this without drowning the team in flaky false failures — the failure mode that kills trust in a test suite and makes people ignore it.

Proposal

Establish a platform standard: every web-app feature ships with Playwright e2e tests that prove its acceptance criteria, authored and owned in-repo by R&D (via the qa-engineer agent), and wired into CI as gates. This is the layer "we ensure exists."

In parallel — not integrated — Testim.io remains an independent suite authored and maintained by Bar (codeless, self-healing, with its own dashboard and scheduling). Aviad wants both; they live side by side. The platform standard does not manage Testim; qa-engineer only coordinates with it to avoid duplicating the same coverage. The CI gating rides on Playwright because it lives in the repo and becomes a natural required status check; Testim runs as parallel regression and does not block our PRs (unless we later decide it should — see open questions).

The governing principle for both suites: quality over coverage. Given a feature's acceptance criteria, choose a small set of high-value scenarios that prove them (the critical path plus material edge cases). Each test is justified by an acceptance criterion and reviewed like code — never machine-generated blanket coverage that no one trusts.

Why now / Why us

The product-intake workspace is about to make acceptance criteria a first-class, consistently-captured input at the PRD level. That is exactly the upstream signal an e2e standard needs: once ACs are reliably written, "a test per AC" becomes mechanical to require and to review. Building the testing standard now means it is ready to consume ACs the moment product-intake lands, instead of retrofitting later. R&D owns the CI, the repos, and the qa-engineer agent, so the in-repo Playwright half is ours to define; the platform (bewith-docs) is where cross-repo standards already live.

Sketch of approach

Trigger model (blocking rides on Playwright):

When	What runs	On red
Before merge	e2e for the area that changed + related area(s)	Blocks the merge
Before deploy	smoke (critical happy paths)	Alert + ask approval to continue — the person merging/deploying decides (a long deploy should not silently auto-fail on this, but they are warned)
Nightly + manual	full suite	Reported to the dashboard

Defining "related area" (a maintained mechanism, not inference magic):

• Tests are tagged by the area they cover (e.g. @payments, @withdrawals).
• A maintained paths -> area map says which source paths belong to which area (changed files under src/payments/** run @payments plus its declared adjacent areas). CI diffs the branch against base, computes touched areas, runs the matching tags.
• Touching a core area (auth, contracts, payments) triggers a broader set.
• Cross-repo effects (a backend change breaking a frontend test) are not the pre-merge target's job — they are caught by the pre-deploy smoke and the nightly full run. Those are the safety nets.
• qa-engineer owns and maintains the area map.

Deterministic test data (against a shared staging environment — this is the flakiness lever):

• A dedicated test tenant plus an idempotent seed (seed:e2e) run before each run, so every run starts from an identical, known baseline. Hard staging-only guard — never prod.
• Each mutating test provisions its own data via the API (not the UI), stamping a unique run-id on every entity it creates (e.g. e2e+RUNID@...), and tears it down after. A periodic sweep reaps orphaned e2e+* leftovers from failed teardowns.
• Ownership: qa-engineer defines the fixture/seed contract; database designs the seed/reset script; backend-developer exposes a guarded staging-only seed endpoint.

Test auth: Cognito passwordless OTP cannot be received in CI, so we need a test-user / token-mint path scoped to the test tenant, owned by auth-security. The existing permissions-demo-tokens skill mints JWTs locally; a staging analog is the target.

Dashboard: the Playwright HTML/JSON report published from CI (nightly/manual). Testim keeps its own dashboard in parallel — no unified dashboard is built now (that would be over-engineering).

Acceptance criteria source: ACs come through product-intake at the PRD level. This standard does not duplicate AC enforcement in the PRD template — that belongs to product-intake. It only consumes ACs. As a fallback when a task has no PRD, an AC section in the issue/task template is added, coordinated with product-intake.

Scope — which repos compose the web application (candidate map, to be confirmed with the team — see open questions):

• Backend: backend-services (NestJS /api/v2).
• Web frontends: bewith-consumer-frontend, management-webapp, organization-dashboard, partner-portal, support-tool-frontend.
• Legacy (deprecation targets — smallest-safe coverage only): superco-consumer (PHP/Yii2), superco-management / superco-management-17 (Angular).
• Explicitly out of scope: mobile (leaders-app, consumer-mobile), libraries (design-system, components), and bewith-docs itself (a static docs site with no app features).

Artifacts this produces (authored later via /new-artifact once the proposal is accepted):

• A policy — a feature does not merge without e2e covering its ACs; red pre-merge e2e blocks the merge; red pre-deploy smoke triggers the approval gate.
• qa-engineer agent update — owns the Playwright standard, the area map, the data/auth/fixture contract, the quality-over-coverage discipline, and Testim-coexistence awareness.
• CI reusable workflow (@ci-v1) extension — per-area pre-merge run, pre-deploy smoke, nightly/manual full run, and publish of the Playwright report/dashboard.
• A methodology doc — the "why" and the two-suite (Playwright + Testim) model.
• An issue/task template AC-section fallback (coordinated with product-intake).

Rollout: phased. Pilot on one repo (a strong candidate: backend-services for API-level e2e, or one self-contained frontend), prove the data/auth/CI spine, then expand repo by repo.

Open questions

• Should Testim also gate, or stay purely parallel (regression + its own dashboard, non-blocking)? This proposal assumes parallel/non-blocking; if Aviad wants Testim to block too, that adds a Testim API/CLI-to-CI integration layer.
• Exact web-app repo list — the scope map above is a candidate; confirm membership and the pilot repo with the team.
• Where the Playwright dashboard is hosted — a CI artifact per run, or a published page (and if published, under the same Private-Pages SSO as bewith-docs)?
• Staging readiness per repo — the backend has an IL staging tier; the frontends' staging URLs and stability need confirming before they can be gated.
• Effort/RFC — given L effort, should this graduate to an RFC before execution?

Related

• proposal-2026-06-14-self-cleaning-ephemeral-verify-harness — the local verify-and-teardown harness; a sibling on the "provision, verify, clean up" theme, but for local dev rather than staging CI e2e. The teardown/data-lifecycle discipline overlaps.
• qa-engineer agent — the owner of the Playwright half.
• product-intake repo — the upstream source of acceptance criteria this standard consumes.
• permissions-demo-tokens skill — the local JWT-minting pattern the test-auth path extends to staging.

---

Triage notes

Filled in during triage. Empty until then.