Visual smoke tests

Using mabl, you can easily create a smoke test that automatically checks important web app and site pages for correct loading, visual changes, broken links, JavaScript errors and network activity issues across browsers. To create such a test, all you need to do is provide mabl with your page URLs and mabl will auto-generate the test for you to run and review the results.

You can improve your test coverage and start seeing results in less than five minutes! .

Visual smoke test results overview

Visual smoke test results overview

A visual smoke test differs from the other mabl tests in the following ways:

  • Detects visual changes for each browser the test runs in while other tests only do so in Chrome.
  • Checks for broken links on each of the specified pages instead of crawling everything
  • Offers a more elaborate workflow to review visual changes, update the baseline and flag regressions for each configured browser type.

Note that having broken links is not only bad for the user experience but it can also hurt your rankings in search engine results (e.g. Google, Bing), especially for your marketing website which should be an essential part of your app and the "whole product experience."

Use cases

The visual smoke test is great for continuous testing across your staging and production environments to validate:

  • Key pages from the customer's journey on your marketing website, such as the home page, product overview and pricing
  • Key landing pages for your marketing website, such as trial signup, demo request, and whitepaper download that you drive traffic through via advertisement
  • Key pages of your web application, such as signup, login, user settings, etc.

Your workflow

  • Identify key pages for your web app and site based on your analytics
  • Create a visual smoke test from the mabl app for those page URLs
  • Run that test manually, on schedule and/or deployment event
  • Review test results and accept, dismiss or flag visual changes
  • Capture and address any issues with the app or website under test

Creating a visual smoke test

Creating a visual smoke test is simple, and it doesn't require using the mabl Trainer. You can click to create a new visual smoke test from two places:

  • The "New" button on the Dashboard page
  • The "New test" button options on the Tests page

You will then be presented with a form where you can enter up to 50 page URLs for mabl to test under a given testing plan. Fill out the form to create the test.

Easily create lists of URLs

In Chrome's Bookmark Manager, you can multi-select bookmarks and copy them and then paste them into your test!

Visual smoke test creation form.

Visual smoke test creation form.

Once you complete the test creation form, you will see the list of page URLs you specified with a mabl-generated description of the test steps. You can edit the page URLs and other test properties, such as description and labels, but you cannot edit the test steps since those are auto-generated for the test based on mabl algorithms.

Visual smoke test details.

Visual smoke test details.

* Using variable in the URLs
To run the same test across multiple environments such as staging and production, you can compose the URLs using the system variable {{@web.defaults.url}} which will use the base URL configured for the environment in mabl.

For example, you can type {{@web.defaults.url}}/product for the URL and this will become https://staging.mabl.com/product when test is run against staging and when run against production the URL will be https://mabl.com/product.

Running a visual smoke test

To run the test, you can use the "Run test" button on the test details page or run it as part of a plan just like all other tests in mabl. When you run the test for the first time, mabl will visit each page URL to:

  • Check it loads properly (200 status code response)
  • Capture a screenshot of the page to establish a fixed visual baseline
  • Check for broken links and JavaScript errors on the page
  • Capture a DOM snapshot with all page network activity for diagnostic purposes

To start seeing visual changes, you need to run a test at least two times since the first run is used to establish a visual baseline. This requirement also applies when you update the page URLs because mabl assumes it needs to create a new visual baseline for the updated URLs.

Insights

You will receive the following insights as a result of a visual smoke test:

  • Visual change detected
  • Broken links
  • JavaScript errors

Those insights will show up in the mabl app under the Insights section as well as in Slack, if you have it configured.

Visual smoke test limitations

  • The viewport cannot be changed and is currently set to 1366 x 3072 for all browsers
  • Visual learning to identify dynamic content regions is not yet supported for this type of test
  • It is not yet possible to manually specify regions to ignore for visual change detection

If you would like us to prioritize addressing these limitations, or want to request additional features, please let us know at feedback.mabl.com.

Reviewing visual smoke test results

When viewing results for the visual smoke test, you will see thumbnails of all page screenshots to the left with the full screenshot of the selected page to the right. The number of detected visual changes and broken links is listed at the top.

Visual smoke test results.

Visual smoke test results.

You will see a yellow camera icon on the thumbnail for each page with visual changes. Selecting it, will allow you to examine the changes in a side-by-side view and take one of the following actions on the current screenshot with visual changes:

  • Accept it as the new baseline for future test runs
  • Dismiss it as reviewed without accepting as a new baseline or flagging it as an issue
  • Flag it as visual regression so it can be fixed

Capturing issues

When you flag a visual change as regression, that flag will remain as part of the test run and you can simply share a link to the test run with your software development team so they can take a look at the results. You can also capture regression issues in Jira directly from the test results page, if you have the integration configured.

To get the list of broken links, simply click on the number of broken links and download the report as a CSV file.

Updated 15 days ago



Visual smoke tests


Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.