Much of automated UI testing relies on finding elements by their text-based attributes, such as selectors and classes. However, sometimes these attributes aren't enough to reliably and accurately interact with certain types of elements, such as canvas elements, PDFs, maps, or elements that lack unique attributes.
With visual find, you can leverage the power of AI to target an element based on its visual characteristics instead of relying on text-based attributes in the DOM or page source. Draw a box around the element, and mabl writes a natural-language description that it uses to locate the element at run time.
Visual find is available for the following step types:
| Test type | Step types |
|---|---|
| Browser tests | Click, enter text, select, create variable (element property), and assert (element) |
| Mobile tests | Tap |
This article explains how to train steps that use visual find.
Visual find vs. visual fallback
A visual find is how the step locates the element in the first place. A visual fallback is an optional that only runs when the step's primary find method fails.
Add a new step
After launching your application in the mabl Trainer, get your application in the correct state and add a new step. How you get to the visual find option depends on the type of step:
| Step type | How to add it |
|---|---|
| Tap (mobile tests) | + (Add step) > Tap |
| Click (browser tests) | + (Add step) > Click |
| Enter text or select (browser tests) | Perform the action in your app so the Trainer records the step, then edit the recorded step. |
| Create variable (browser tests) | + (Add step) > Create variable > Element property |
| Assert (browser tests) | + (Add step) > Assert > Element |
Add click step
Trainer only
Drawing a box requires the live page, so the Visual area option is only available in the mabl Trainer. If you're editing a test with quick edit in the web app, open the test in the Trainer to use visual find.
Select the target area
For tap and click steps, the step configuration menu selects Visual area by default. For all other step types, select Visual area under How to find the element. Then click and drag to draw a box over the area you want to target.
Targeting an element by visual area for an assertion
mabl automatically sends a screenshot of the target area to the model, which sends back an AI-generated description of the selected area. mabl performs the find using the GenAI description to confirm that it finds a match.
For mobile tests, the image preview shows where mabl will perform the tap step in the selected area on execution. If you need to target a precise location within the selected area, you can adjust the x-y coordinates.
Previewing coordinates
Review the description
If the description fails to highlight the correct element, reselect the area or manually edit the description and test the find again.
Even if the description highlights the correct element, you should also ensure the description has the correct intent. Depending on what you're trying to achieve, the model might generate a description that accurately highlights the correct element during training, but doesn't capture your ultimate intent.
To illustrate, consider an app with many grid elements.
Example targeting an element on a grid
The model generated the description "A turquoise rounded square tile with the text 'Car 10' in the center", which highlights the correct element. However, if the intent is to target a tile at a specific position on the page, you should manually update the description to reflect this intent - "The second square from the left, third row from the top" - and test it out.
Choose a property or attribute
For create variable and assert steps, the rest of the form reads the element that your description resolves to, so you pick from values the element actually has rather than a fixed list. Select Test description first so mabl can resolve the element; until you do, the form asks you to run the description rather than offering a value it can't verify.
- Create variable > Element property: the Property dropdown lists the attributes and properties of the resolved element, and the preview shows its current value on the page.
- Assert > Element: the HTML attribute and CSS property pickers list the resolved element's real attributes and computed styles.
Selecting an element property for a create variable step
Save the step
When you are satisfied with the step, click Save to add it to your test.
Running tests with visual find
On execution, mabl sends the GenAI description and the runtime screenshot back to the model, which returns the target visual area, known as a bounding box. mabl then attempts to resolve an element within that box and performs the step action on it. For taps and clicks, mabl only falls back to clicking the center of the bounding box if no interactable element is found.
Step output for a visual find step