E2e reliability and performance#2201
Draft
samuelreichert wants to merge 20 commits intomainfrom
Draft
Conversation
github-actions
Bot
added
data-widgets
accordion-web
badge-button-web
badge-web
carousel-web
charts-web
color-picker-web
image-web
language-selector-web
maps-web
popup-menu-web
progress-bar-web
progress-circle-web
range-slider-web
rating-web
rich-text-web
slider-web
switch-web
timeline-web
tooltip-web
video-player-web
accessibility-helper-web
fieldset-web
automation
workflows
combobox-web
markdown-web
labels
Introduces reusable test infrastructure: - fixtures.mjs: custom Playwright test with auto Mendix readiness wait and guarded session cleanup (replaces 54 manual afterEach blocks) - mendix-helpers.mjs: waitForMendixApp, waitForWidget, waitForListData, safeLogout, navigateToPage utilities Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…defaults - Add actionTimeout (10s) and navigationTimeout (30s) to catch hangs - Add global screenshot defaults: animations disabled, threshold 0.1 - Replaces per-test threshold overrides with a sensible global default Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace cypress/**,cypress.config.* with e2e/**,playwright.config.* so turbo correctly tracks E2E test file changes for cache invalidation. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add eslint-plugin-playwright rules for E2E spec files: - no-wait-for-timeout (error): blocks new hardcoded delays - no-networkidle (warn): flags unreliable networkidle usage - prefer-web-first-assertions (warn): encourages auto-retrying assertions Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
migrate-spec.mjs automates three transforms per spec file:
1. Replace @playwright/test import with shared fixtures
2. Remove afterEach logout blocks (fixture handles cleanup)
3. Replace waitForLoadState("networkidle") with waitForMendixApp
Supports --dry-run flag for preview. Run per file or batch via find.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- column-chart: wait for .plot-container visibility instead of 1000ms - datagrid-dropdown-filter: assert row count instead of 300ms delay - gallery: assert item count after filter instead of 1000ms delay - heatmap-chart: remove 500ms delay (colorbar visibility already asserted) - skiplink: use toHaveCSS assertion instead of 1000ms + manual evaluate Eliminates all 12 hardcoded timeout calls from E2E specs. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Run codemod across 55 E2E spec files to:
- Replace @playwright/test import with @mendix/run-e2e/fixtures
- Remove manual afterEach session logout (fixture handles cleanup)
- Replace waitForLoadState("networkidle") with waitForMendixApp()
The shared fixture provides automatic Mendix readiness detection and
guarded session teardown, eliminating 54 manual logout blocks and
154 unreliable networkidle waits.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…rides Remove all per-test threshold (0.1-0.5) and maxDiffPixels (4000-15000) overrides. Global config now sets threshold: 0.1 and animations: disabled. - maps-web/google: replace meaningless 15000px-diff screenshot with structural assertion (canvas/div visibility) - maps-web/here,mapbox,openstreet: remove maxDiffPixels: 4000 - charts (pie, heatmap, time-series, column): remove threshold: 0.5 - rich-text, slider, timeline: remove threshold: 0.2-0.4 Screenshot baselines will need regeneration on CI after this change. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace sequential --workspace-concurrency=1 execution with 4 parallel runners using run-e2e-in-chunks.mjs bin-packing distribution. Expected ~75% reduction in nightly E2E wall-clock time. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds checkAccessibility(page, selector, options) to mendix-helpers.
Wraps @axe-core/playwright with sensible defaults (wcag21aa tags).
Usage:
import { checkAccessibility } from "@mendix/run-e2e/mendix-helpers";
await checkAccessibility(page, ".mx-name-widget1");
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Without explicit exports, Node cannot resolve @mendix/run-e2e/fixtures or @mendix/run-e2e/mendix-helpers from widget spec files. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…idle The previous implementation resolved too early — mx.session exists before sub-page widgets render. Now waits for: 1. domcontentloaded 2. mx.session + no progress indicator + .mx-page exists 3. networkidle (ensures widget data fetches complete) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…der)
Codemod regex only matched `{ test, expect }` ordering. Six specs used
`{ expect, test }` and were left importing from @playwright/test — no
fixture cleanup meant session exhaustion after ~5 tests.
Also fix codemod regex to handle both orderings for future runs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Make mendixSession worker-scoped so each worker holds 1 Mendix session, preventing session exhaustion with parallel workers (4 workers < 5 license limit). - Fix waitForFunction call: pass timeout as 3rd arg (options), not 2nd (arg). Previously actionTimeout (10s) was used instead of intended 60s. - Set local workers to 4 (safe with worker-scoped sessions). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- FilteringSingle: replace allTextContents() after toHaveText() with single toHaveText([...array]) call that retries atomically. - FilteringMulti: remove waitForTimeout(300) and allTextContents() pattern, use toHaveText()/toContainText() for auto-retry assertions. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…specs - video-player-web: remove redundant waitForMendixReady (fixture handles it). - checkbox-radio-selection-web: replace networkidle waits with waitForMendixApp. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ening Regenerated with threshold 0.1 (previously 0.4-0.5). Minor layout shifts in row heights and selection highlighting. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…itForDataReady networkidle hangs forever on pages with streaming content (video embeds, websockets). The mx.session + .mx-page check is sufficient for readiness. Add waitForDataReady helper for tests that explicitly need networkidle. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… baselines New local baselines for macOS development. CI uses chromium-linux variants. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
samuelreichert
force-pushed
the
e2e-reliability-and-performance
branch
from
05040a8 to
e7a39b8
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull request type
Description