Failure reasons

When testing complex applications with many parts, it's important to see the big picture. In mabl, you can add a failure reason to failed test runs to help your team understand why your tests are failing and the state of your overall development.

This article describes use cases for failure reasons and how to add failure reasons to a test run.

📘

View historic test data

The best way to access historic test data is with the BigQuery export integration, which gives you the ability to report on all your test runs from one place. With this integration, you can also view all of your categorized failures from the time you set it up.

Use cases

Use cases for adding failure reasons to failed test runs include:

Understand overall application quality

The more you and your team mark the reason for failed tests, the more you'll get an idea of your overall application's quality. Failure reasons are reported from the dashboard on the mabl home page and the release coverage dashboard.

For example, if most of your failed tests are categorized as test implementation issues, you may want to try out branching to make sure that your regression or smoke tests aren't affected by new features added to your app.

📘

Ad hoc runs

Ad hoc test runs, which are test runs without a plan, can be classified with a failure reason. However, they are excluded from the "Top failure reasons" dashboard chart as well as the "Test run history" chart.

Filtering recently failed runs

On the results page, you can filter recent runs by failure category and export these results via the Download CSV button above the filters.

Save time on review

When you add a failure reason to a failed run, you help your teammates spend less time trying to understand why things went wrong.

If someone on your team added a failure reason to your test, you can hover over the failure reason on the Test Output page to find out who changed the failure reason and when they changed it.

To find out which test failures are not yet categorized by your team, go to the Results page and set the "Failure reason" filter to "Uncategorized."

How to add a failure reason

On the Test Output page of a failed test run, click on the "Add failure reason" dropdown. Failure reasons include the following:

Regression

This failure reason means that mabl has caught a bug that caused your test to fail, such as a button disappearing after a recent release or a popup appearing where it shouldn't have previously.

Environment issue

Environment issues are any failures caused by something local to your testing, development, or other environments, such as dev credentials that are no longer valid or an environment suddenly becoming private.

Network issue

Network issues are any failures that may be related to mabl failing to connect to your app.

Test implementation issue

Test implement issues failures are related to how the test was originally trained. For example, a failed test with a test implementation issue could be caused by recording steps in the wrong order or accidentally deleting an important step.

Timing issue

Timing issues are failures related to the performance of your application, such as an application failing to load an element on the page in time for mabl to interact with it. We recommend using a wait until step in that situation to make your tests more robust.

Other issue

This failure reason is for anything that might not fit into the categories above.

Mabl issue

This failure reason is for any failures that you believe are related to how mabl is executing your tests. If you believe a test failed due to a mabl issue, we recommend also reaching out to the mabl support team.

Removing failure reason

To remove the failure reason, click the X on the dropdown. Removing a failure reason counts as an edit, so if you hover over the dropdown, you can see the last person to remove a failure reason.