When a visual smoke test runs in the cloud, the test output includes the following:
- Screenshots and logs: Similar to browser test output, visual smoke tests update the test output page with screenshots and logs in real time.
- Visual changes: Mabl processes each screenshot to detect visual changes. If mabl detects a visual change for a particular page in the test, the visual change appears on the test output page.
While mabl is still calculating visual changes, a banner appears on the test output page: "Scanning for visual changes; refresh for latest results".
Visual smoke test output
When a visual smoke test runs in the cloud, it counts as one browser test run.
Screenshots and logs
Every time a visual smoke test runs in the cloud, the mabl-generated flow "Visual Smoke Test" visits each URL and completes the following actions:
- Capture a screenshot of the page
- Collect test diagnostics:
- Check that the page returns a 200 response
- Capture network activity on the page
- Capture a DOM snapshot
It takes two runs for mabl to identify visual changes in a visual smoke test.
- On the first plan run, mabl establishes a visual baseline.
- On subsequent runs, mabl compares each page to its visual baseline.
The comparison to the visual baseline is a static comparison between the current run and the visual baseline. Each pixel is compared and considered a match if the exact same color is in the the exact pixel location of both screenshots. Then, differences are clustered and related to the full DOM information captured during the runs. Finally, components encompassing regions of individual pixel changes are identified and marked as visual changes.
If the differences between the two screenshots meet the minimum visual change threshold, which is roughly the size of a blinking cursor, mabl does the following:
- Highlight the regions of the page that appear different
- Flag the step as a visual change in the test output
- Report a visual change insight in your workspace
Mabl create a separate visual baseline for each browser that the test runs on.
If you make any updates to the list of URLs to visit, mabl establishes new visual baselines for all pages on the next plan run.
Reviewing visual smoke tests
When mabl generates visual change insights for a visual smoke test, you can follow this workflow for reviewing visual smoke test output:
- Identify visual changes
- Choose a review action
- Escalate flagged steps
- Resolve flagged steps
Identify visual change
On the insights page, click on the "New visual changes" link for the test that you would like to review.
The insights page
Click on the RESULTS link to view test output.
Visual change insights
You can filter the output to show only steps with visual changes. Steps with visual changes have an orange camera icon.
Test output page for a visual smoke test
Click on the step to compare screenshots. The screenshot tab has three views:
- Current: screenshot from the current run
- Baseline: screenshot from the run in which mabl established the visual baseline
- Side-by-side: comparison view of the current and baseline screenshots
Choose a review action
For each step with visual changes, choose one of the following actions from the Review dropdown:
- Accept as new baseline: in future plan runs the current screenshot becomes the visual baseline.
- Dismiss as reviewed: maintain the existing visual baseline.
- Flag as visual regression: add a flag to the step to indicate that the page needs to be investigated.
Escalate flagged steps
Flagging a step identifies it for your team as a change that needs further review. Some options for escalating flagged steps in visual smoke tests include:
- Share a link to the test run with your team so that they can look at the results. If you are working in the Desktop App, follow the steps on sharing links outlined here.
- If the Jira integration is set up for your workspace, create a new issue from the test output page.
Resolve flagged steps
If your team reviews flagged steps for visual regressions and finds that the visual changes in the flagged step are expected, they can accept the new visual change: Review > Accept as new baseline.
If the visual changes in the flagged step indicate a visual regression, your team can fix the issue in the app and then dismiss the step in the visual smoke test as reviewed: Review > Dismiss as reviewed.