Creating visual assertions

With visual assertions, you can validate more complex aspects of your application, such as text content, quality, and images with generative AI. 

This article explains how to create visual assertions to validate page content in web and mobile applications. For more information on using visual assertions to validate the contents of downloaded files, see the article on download assertions. 

How to create a visual assertion

To create a visual assertion, open your web or mobile application in the mabl Trainer and get your application in the desired state. Then take the following steps:

1. In the Trainer window, click on the check mark icon to create a new assertion.
2. Identify what you want to validate:
   • To validate the entire page, click on Screen.
   • To validate part of the page, click on Element. Select the element you want to validate on the page and select Visual in the element assertion menu.
3. In the Assertion description field, describe what you want to verify using natural language. Your assertion description may include references to variables used in the test, including values from DataTables and other data-driven variables. For example, “Verify that the results are relevant to {{@search_input}}.”
4. Click on Test assertion. The first time you test the assertion, mabl generates assertion criteria.  This criteria is sent with the assertion description to the LLM to provide a consistent set of criteria to evaluate as part of your assertion. If you iterate on the assertion description, mabl updates the assertion criteria to reflect the current state of the description.
5. Choose a behavior on failure. By default, visual assertions are set to “Fail at end of test.”
6. Click OK.

For more guidance on writing an assertion description, see the article on best practices for visual assertions.

Visual assertions and viewport size

Unlike HTML and CSS assertions, visual assertions evaluate based on the screenshot of the element, not the DOM or the page source.

• For a visual screen assertion, the screenshot only includes the portion of the page that is currently visible in the screenshot.
• For a visual assertion on a specific element, the screenshot only includes the portion of the element that is visible in the viewport.

If you assert on something that isn’t fully visible in the viewport, such as a long table element, the off-screen part of the element will not be evaluated.

To resize the window in a browser test, add a “set viewport” step before the assertion. The “set viewport” step ensures that the test uses the new viewport size for the visual assertion during test execution. If you manually resize the browser window during a training session, the browser will still use the size specified in the “set viewport” step when run in the cloud.

Returning output in non-English languages

The models used for visual assertions are biased towards returning results in English. If you submit a prompt in a non-English language, you can increase the likelihood of returning the result in the same language by specifying the desired output language.

For example, to return results in Japanese, add the sentence, “Return the results in Japanese” to the end of the prompt.

Running tests with visual assertions

visual assertions are supported in Trainer replay, local, and cloud execution. When triggering a local run from the mabl CLI, you must use the flag --allow-billable-features to enable visual assertions. For example:

mabl tests run --id {test-id} --allow-billable-features

Execution limitations

During training, visual assertions are unlimited so that you can iterate on your assertion as much as you need. When executing tests locally or in the cloud, however, keep in mind the following limitations:

• Limit per test: visual assertions are limited to 30 per test.
• Performance tests: visual assertions will fail when run within the context of a performance test.