We are pleased to share that GenAI Assertions are now generally available! GenAI Assertions are great for validating more complex aspects of your web or mobile application, such as text content, linguistic analysis, and visual details.
GenAI Assertions promotional program
Through the end of 2024, customers and trial users can run tests with GenAI Assertions as part of a special promotional program:
- Customers may execute up to 1000 cloud runs containing GenAI Assertions.
- Trial users may execute up to 100 cloud runs containing GenAI Assertions.
If the limit is exceeded before 12/31/24 or the promotional program has ended without the inclusion of the feature in your mabl plan, the feature will be disabled. Any tests or plans containing GenAI assertions cannot be run until those steps are removed. If the test is executed as part of a deployment event, the test containing GenAI assertions will be skipped.
Try it out
To try it out for yourself, launch your web or mobile application in the mabl Trainer and follow these steps to create a GenAI Assertion.
Since introducing GenAI Assertions in June, we've updated the behavior in the Trainer to include assertion criteria. This assertion criteria is sent with the assertion description to the LLM to provide a consistent set of criteria to evaluate as part of your assertion. If you iterate on the assertion description, mabl updates the assertion criteria to reflect the current state of the description.
With the GA release, GenAI Assertions now capture explicit criteria in the assertion step for greater consistency. These newly added or edited GenAI assertions are not compatible with older versions of the Desktop App. Please make sure that your team updates to mabl Desktop App version 2.18.0 or higher to continue working with any tests that include GenAI Assertions.
Execution limits
GenAI Assertions are unlimited during training so that you can iterate on your assertion as much as you need. During execution, however, keep in mind the following limitations:
- Cloud runs: GenAI Assertions are limited to 6 per test. If they exceed this limit, the test will fail.
- CI Runner and CLI local runs: Any GenAI Assertions will fail when run locally in the mabl CLI or the CI Runner.
- Performance tests: GenAI Assertions will fail when run within the context of a performance test.
Learn more
To learn more about GenAI Assertions, check out the following articles:
Protecting customer data
Mabl’s generative AI capabilities are built on top of Google Cloud’s enterprise AI tools. Neither mabl nor our service partner, Google Cloud, use customer data for training these models. If you have any concerns about the use of generative AI in mabl, please reach out to your customer success manager.