When testing complex applications with many parts, it's important to see the big picture. In mabl, you can add a failure reason to failed test runs to help your team understand why your tests are failing and the state of your overall development.
Adding a failure reason to a failed test run
This article describes use cases for failure reasons and how to add failure reasons to a test run.
View historic test data
The best way to access historic test data is with the BigQuery export integration, which gives you the ability to report on all your test runs from one place. With this integration, you can also view all of your categorized failures from the time you set it up.
Use cases
Use cases for adding failure reasons to failed test runs include:
Understand overall application quality
The more you and your team mark the reason for failed tests, the more you'll get an idea of your overall application's quality. Failure reasons are reported from the dashboard on the mabl home page and the release coverage dashboard.
Top failure reasons reported on the dashboard on the mabl home page
For example, if most of your failed tests are categorized as test implementation issues, you may want to try out branching to make sure that your regression or smoke tests aren't affected by new features added to your app.
Ad hoc runs
Ad hoc test runs, which are test runs without a plan, can be classified with a failure reason. However, they are excluded from the "Top failure reasons" dashboard chart as well as the "Test run history" chart.
Filtering recently failed runs
On the results page, you can filter recent runs by failure category and export these results via the Download CSV button above the filters.
Save time on review
When you add a failure reason to a failed run, you help your teammates spend less time trying to understand why things went wrong.
If someone on your team added a failure reason to your test, you can hover over the failure reason on the test output page to find out who changed the failure reason and when they changed it.
Finding out who added a failure reason
To find out which test failures are not yet categorized by your team, go to the results page and set the "Failure reason" filter to "Uncategorized."
How to add a failure reason
On the test output page of a failed test run, click on the Add failure reason dropdown. Failure reasons include the following:
Regression
This failure reason means that mabl has caught a bug that caused your test to fail, such as a button disappearing after a recent release or a popup appearing where it shouldn't have previously.
Environment issue
Environment issues are any failures caused by something local to your testing, development, or other environments, such as dev credentials that are no longer valid or an environment suddenly becoming private.
Network issue
Network issues are any failures that may be related to mabl failing to connect to your app.
Test implementation issue
Test implement issues failures are related to how the test was originally trained. For example, a failed test with a test implementation issue could be caused by recording steps in the wrong order or accidentally deleting an important step.
Timing issue
Timing issues are failures related to the performance of your application, such as an application failing to load an element on the page in time for mabl to interact with it. We recommend using a wait until step in that situation to make your tests more robust.
Accessibility issue
Accessibility issues indicate that the test did not meet the criteria of an accessibility check.
Performance issue
Performance issues represent tests that failed because of performance issues in the testing environment or performance tests that failed because they met the failure criteria as defined by the test.
Other issue
This failure reason is for anything that might not fit into the categories above.
mabl issue
This failure reason is for any failures that you believe are related to how mabl is executing your tests. If you believe a test failed due to a mabl issue, we recommend also reaching out to the mabl support team.
Removing failure reason
To remove the failure reason, click the X on the dropdown. Removing a failure reason counts as an edit, so if you hover over the dropdown, you can see the last person to remove a failure reason.
Finding out who deleted a failure reason