We are thrilled to share that GenAI Assertions are now available for mobile tests in early access!
Use GenAI assertions to expand your test coverage and validate more complex aspects of your mobile application, particularly textual and visual details that aren't easily validated using the page source.
GenAI Assertions are currently available for all mabl customers as an early access program. Trial users can reach out to the mabl team if interested in trying it out.
Prior to general availability, changes to the feature may be made without notice.
How it works
A GenAI Assertion is a type of assertion that sends a screenshot of the target element and a description of what you want to validate to a large language model (LLM). The LLM determines whether the assertion should pass or fail in the current situation and returns a assertion result with an explanation of why the assertion passed or failed.
Try it out
If you have already opted in to GenAI Assertions for browser tests, you will automatically be opted in to GenAI Assertions for mobile tests.
If your team hasn't yet opted in, an your workspace can enable GenAI Assertions on the Labs page: Settings > Labs. They may need to update to the latest version of the mabl Desktop App to see the option on the labs page.
To create a GenAI Assertion, open your mobile application in the mabl Trainer and get your application in the desired state. Then take the following steps:
- In the Trainer window, click on the check mark icon to create a new assertion.
- Select the element on the page that you want to validate.
- Click on the AI tab in the assertion menu.
- In the Assertion description field, describe what you want to verify.
- Click on Test assertion.
- Iterate on the assertion description until you are confident that the assertion yields the desired result.
For more guidance on creating successful GenAI assertions, see our article on creating assertions with generative AI.
GenAI Assertions evaluted based on the screenshot of the element, not the page source. If needed, add a scroll to element step to make sure the screen shows the correct area. The screenshot included with a GenAI Assertion only includes the portion of the element that is currently visible in the viewport.
Limitations
GenAI Assertions are not supported for:
- Local test runs triggered in the mabl CLI
- Tests that execute as part of a performance test run.
The models are biased toward prompts written in English. Prompts written in other languages are not guaranteed to yield a result in the same language.
During the early access period, keep in mind the following limitations:
- GenAI Assertions are limited to 3 per cloud run.
- If you're running plans or deployments in parallel, no more than 20 of the parallel test runs should include GenAI Assertions. Exceeding this limit may result in the feature being disabled for the workspace.
To avoid exceeding these limits, we recommend against using GenAI Assertions in login flows, loops, and other commonly used flows.
GenAI Assertions are built on top of Google Cloud's AI tools. Neither mabl nor our service partner, Google Cloud, use customer data for training these models. If you have any concerns about the use of generative AI in mabl, please reach out to your customer success manager.
For more information about how mabl handles customer data, see the article on data encryption.