If you need to validate more complex aspects of your application, try creating assertions with generative AI. A GenAI Assertion sends an element screenshot and the assertion description to a large language model to determine whether the assertion should pass or fail.
This article outlines common use cases and limitations of validating your application with generative AI. For step-by-step guidance on how to create a GenAI Assertion, click here.
GenAI Assertion in a mobile test
Protecting customer data
mabl’s generative AI capabilities are built on top of Google Cloud’s enterprise AI tools. Neither mabl nor our service partner, Google Cloud, use customer data for training these models. If you have any concerns about the use of generative AI in mabl, please reach out to your customer success manager.
Common use cases
GenAI Assertions are great for evaluating the content and quality of text and images in your application that are difficult or impossible to validate with traditional HTML or CSS assertions.
For additional guidance on crafting a prompt that yields valid and reliable results, see the article on best practices for GenAI Assertions.
Validating text content
| Use case | Sample prompt |
| Verify the presence or absence of certain words or phrases, including variable values. | “Verify that {{@label}} is visible in one or more rows of this table.” |
| Verify that the page is in the correct language. | “Confirm that all the text on the page is in Spanish.” |
| Check for correct grammar and spelling. | “Verify that the blog post has no grammatical errors or typos.” |
| Verify proper formatting, such as bold or hyperlinked text. | “Verify that the privacy policy is a hyperlink.” |
| Validate multiple aspects of text and visual context in one assertion | “Confirm the following are present in the screenshot: - The mabl logo - A button to start a free trial - A language picker - Also validate that there are no broken images.” |
Validate text quality
| Use case | Sample prompt |
| Verify that the results are correct given a specific input. | “Confirm that the results on the page are relevant to the {{@search_input}}.” |
| Verify that the response from a chatbot is appropriate given a specific input. | “Confirm that the chatbot response is relevant to the {{@question_input}}.” |
| Verify the tone of text | “Verify that the response from the chatbot uses a professional tone.” |
Validate images
| Use case | Sample prompt |
| Check for the presence or contents of an image, such as colors or objects | “Confirm that there is an image of a globe with Greek characters in it.” |
| Verify that an image is correct given a specific input. | “Confirm that all photos show an image that is relevant to {{@search_input}}.” |
| Create moment-in-time assertions for video content. | “Verify that the bottom of the video shows a pause button, a settings cog, a volume controller, and an icon to make it full screen.” |
Limitations
Some use cases are supported by GenAI Assertions better than others. The following types of GenAI Assertions are not well supported or need additional context:
Sorted tables
If you ask that a table is sorted, the models used for GenAI Assertions have a tendency to lean toward “strict” ascending or descending order. If there are duplicate values in the table, the assertion will fail. In this situation, you should write a more explicit assertion description that specifies that the sorting can include duplicate values.
Math
Generative AI can make mistakes in assertions that require math, such as Make sure all rows add up to less than X or Make sure the value in the total column equals the sum of columns A.
Visual overlap
The GenAI model doesn’t perform well in assertions that require an understanding of overlapping or partially obscured elements, such as “Is the text on this page partially covered?”
Spatial arrangement
GenAI models don’t perform well at interpreting prompts to analyze the spatial arrangement of a page, such as whether all the images are centered or whether the items on the screen are in a grid view or a list view.