To expand on previously unsupported use cases, we’re happy to introduce visual find for click steps!
When you record click steps in a browser test, mabl collects information on the target element’s text-based attributes. This method works well for most page elements, but if you’re interacting with a highly visual element, such as a map, canvas element, or image, these text-based attributes may not provide enough information to accurately target the correct element. Visual find uses natural language descriptions to address this gap.
Some examples of how you can use visual click steps include:
- Clicking on a specific data point in a chart to drill into the data
- Interacting with visual icons that lack unique text-based attributes, such as SVGs
- Clicking a pin on a map to validate that an address appears
Try it out
Make sure to update to mabl Desktop App version 2.37.5 and enable Visual find for click steps on the Labs page: Settings > Labs.
After launching your web application in the mabl Trainer, get your application in the correct state and add a step: + (Add step) > Click.
Drag your mouse over the area you want to target. mabl will send a screenshot of the target area to the model, which will send back an AI-generated description of the target area.
Review the generated element description to make sure it has the correct intent. Modify the description as needed and, when you’re satisfied, save the step.
On execution, mabl sends the GenAI description and the screenshot at run time back to the model, which returns the bounding boxes and performs the click action.
Limitation
Currently, visual click steps cannot automatically scroll to a target element that is outside of the viewport. For example, mabl cannot perform a visual click on a map that is at the bottom of the page unless it is already visible when the step runs.
As a workaround, you can record a click or assertion on a nearby element. This action forces the test to automatically scroll to the correct area before the visual click step executes.
Share your feedback
Visual find for click steps is available in mabl Labs: Settings > Labs. If you try it out, we highly encourage you to submit feedback in the mabl Product Portal. Your perspective helps us make sure we build a better product for you and our community!
Learn more about Visual Assist
Visual find for click steps is part of a new suite of find capabilities known as Visual Assist. Visual Assist features use generative AI to understand what UI elements look like to improve find reliability and expand supported use cases. To learn more, check out the following articles:
- Additional languages for element find descriptions
- Visual find for mobile tap steps
- Automatic element description generation for mobile tap steps
Stay tuned for more exciting Visual Assist releases coming soon!