In our ongoing commitment to deliver a more intuitive and intelligent interface for test creation, we're happy to share that assertions powered by generative AI are available in early access! With GenAI Assertions, you can expand your test coverage to validate more complex aspects of your application, including both text and visual details.
AI assertions are currently available for all mabl customers as an early access program. Trial users can reach out to the mabl team if interested in trying it out.
Prior to general availability, changes to the feature may be made without notice.
Read on for more information on making assertions with generative AI:
Try it out
Before you start, make sure to update to the latest version of the mabl Desktop App: Help > Check for updates. During the early access period, workspace owners can enable GenAI Assertions on the Labs page: Settings > Labs.
To create a GenAI Assertion, open your web application in the mabl Trainer and get your application in the desired state. Then take the following steps:
- In the Trainer window, click on the check mark icon to create a new assertion.
- Select the element on the page that you want to validate.
To target the entire page or a specific element, press [SHIFT] and use the up and down arrow keys to scroll through overlapping elements until the Trainer highlights the target element. Release the shift key before clicking on the target element.
- Click on the AI tab in the assertion menu.
- In the Assertion description field, describe what you want to verify. See the following section writing good assertions for tips and examples.
- Click on Test assertion. Ensure that the assertion evaluates the right things, and iterate on the assertion description until you have the desired result.
- Choose a behavior on failure. By default, AI assertions are set to "Fail at end of test."
- Click OK.
When you run a GenAI Assertion, the Trainer combines the assertion description with general information about assessing assertions, the expected response format, and a screenshot of the element. The generative AI model evaluates this input to determine whether the assertion should pass or fail in the current situation.
The screenshot included with a GenAI Assertion only includes the portion of the element that is currently visible in the viewport.
- If you assert on an element that isn't fully visible in the viewport, such as a long table, the bottom half of the element is not included in the prompt.
- To resize the window for a GenAI Assertion, add a "set viewport" step.
In cloud execution, you can review the results of GenAI Assertions in the mabl Activity tab of the test output logs for a given step. Since generative AI is non-deterministic in nature, the results of a GenAI Assertion can vary.
Test execution logs for a GenAI Assertion
How to write good assertions
To increase the likelihood that your GenAI Assertion behaves as you expect it to, keep in mind the following tips:
- Be concise but clear about the intent of the assertion.
- Use simpler, common wording.
- Try to express the idea more generally and include a few examples.
- Avoid too many specific criteria or complex combinations of criteria that can be difficult to evaluate correctly. We recommend breaking a complex assertion with many different cases or criteria into multiple assertions.
Sample GenAI Assertions
To give you an idea of what GenAI Assertions can accomplish, consider the following sample assertions:
Text content and linguistic analysis
- Verify that this page does not have any English text. It should all be Japanese.
- Verify that this string is the correct Spanish translation for X.
- Verify that every item in the list has a description.
- Verify that the tone of the social media post is professional.
Visual characteristics
- Verify that the header containing "Welcome to mabl!" is above the "Start free trial" button.
- Verify that the Reject button looks enabled.
- Verify that all list items are left justified.
- Verify that the rows that indicate out-of-stock items are grayed out.
Limitations
During the early access period, GenAI Assertions are limited to 3 per test. If you're running plans or deployments in parallel, no more than 20 of the parallel test runs should include GenAI Assertions. Exceeding this limit may result in the feature being disabled for the workspace. To avoid exceeding this amount, we recommend against using GenAI Assertions in login flows, loops, and other commonly used flows.
GenAI Assertions are initially supported for browser tests only. At a later date, we will support GenAI Assertions in mobile tests.
GenAI Assertions are not supported for:
- Local test runs triggered in the mabl CLI
- Tests that execute as part of a performance test run.
The models are biased toward prompts written in English. Prompts written in other languages are not guaranteed to yield a result in the same language.
Submit feedback
During the early access period, we highly encourage you to submit feedback. Use the Feedback button in test output logs or share your feedback in the mabl Product Portal. Your perspective helps us make sure we build a better product for you and our community!
Submitting feedback on a GenAI Assertion
We recognize that some organizations have limitations and policies around the use of generative AI. These features are built on top of Google Cloud's AI tools, whose terms of service do not allow use of that data to train Google's general AI models. Furthermore, our generative AI features do not train on, or use, customer data across workspaces. If you have any concerns about these changes or the use of generative AI in mabl, please reach out to your customer success manager about next steps.