Visual smoke test output

When a visual smoke test runs in the cloud, the test output includes the following:

• Screenshots and logs: Similar to browser test output, visual smoke tests update the Test Output page with screenshots and logs in real time.
• Visual changes: Mabl processes each screenshot to detect visual changes. If mabl detects a visual change for a particular page in the test, the visual change appears on the Test Output page.

While mabl is still calculating visual changes, a banner appears on the Test Output page: Scanning for visual changes; refresh for latest results.

📘

Running visual smoke tests

When a visual smoke test runs in the cloud, it counts as one browser test run.

Screenshots and logs

Every time a visual smoke test runs in the cloud, the mabl-generated flow "Visual Smoke Test" visits each URL and completes the following actions:

• Capture a screenshot of the page
• Collect test diagnostics:
  • Check that the page returns a 200 response
  • Capture network activity on the page
  • Capture a DOM snapshot

Visual changes

It takes two runs for mabl to identify visual changes in a visual smoke test.

• On the first plan run, mabl establishes a visual baseline.
• On subsequent runs, mabl compares each page to its visual baseline.

The comparison to the visual baseline is a static comparison. If the page in the current run is not identical to its visual baseline, mabl does the following:

• Highlight the regions of the page that appear different
• Flag the step as a visual change in the test output
• Report a visual change insight in your workspace

📘

Updating visual smoke tests

If you make any updates to the list of URLs to visit, mabl establishes new visual baselines for all pages on the next plan run.

Reviewing visual smoke tests

When mabl generates visual change insights for a visual smoke test, you can follow this workflow for reviewing visual smoke test output:

1. Identify visual changes
2. Choose a review action
3. Escalate flagged steps
4. Resolve flagged steps

Identify visual change

On the Insights page, click on the "New visual changes" link for the test that you would like to review.

Click on the RESULTS link to view test output.

You can filter the output to show only steps with visual changes. Steps with visual changes have an orange camera icon.

Click on the step to compare screenshots. The screenshot tab has three views:

• Current: screenshot from the current run
• Baseline: screenshot from the run in which mabl established the visual baseline
• Side-by-side: comparison view of the current and baseline screenshots

Choose a review action

For each step with visual changes, choose one of the following actions from the Review dropdown:

1. Accept as new baseline: in future plan runs the current screenshot becomes the visual baseline.
2. Dismiss as reviewed: maintain the existing visual baseline.
3. Flag as visual regression: add a flag to the step to indicate that the page needs to be investigated.

Escalate flagged steps

Flagging a step identifies it for your team as a change that needs further review. Some options for escalating flagged steps in visual smoke tests include:

• Share a link to the test run with your team so that they can look at the results. If you are working in the Desktop App, follow the steps on sharing links outlined here.
• If the Jira integration is set up for your workspace, create a new issue from the Test Output page.

Resolve flagged steps

If your team reviews flagged steps for visual regressions and finds that the visual changes in the flagged step are expected, they can accept the new visual change: Review > Accept as new baseline.

If the visual changes in the flagged step indicate a visual regression, your team can fix the issue in the app and then dismiss the step in the visual smoke test as reviewed: Review > Dismiss as reviewed.