Maintaining a healthy test suite can be challenging, especially as your test library grows. To help you prioritize your maintenance efforts for the maximum impact, we’ve added a new test quality table to the coverage dashboard in the mabl app: Coverage > Overview.
The table introduces a quality score for each test: a composite score from 0 to 100 that combines pass rate, stability, and reliability. A low score in any one of these dimensions pulls the overall score down, so you can quickly spot tests that need maintenance.
How it works
Here’s how the three scoring dimensions work:
- Pass rate: the percentage of plan runs where all attempts of the test passed.
- Stability: how consistently the test produces the same result across plan runs. A test with low stability indicates a flaky test that is alternating between passes and failures.
- Reliability: how reliably the test stays passing between consecutive runs. A test with high stability but low reliability is consistently failing, which is a sign that it’s broken rather than flaky.
The overall quality score gives each of these dimensions different weights, and a low score in any of the three dimensions can pull down the score significantly. For example, a test with a 50% reliability rate can pull down the quality score significantly, since 50% really isn't reliable at all.
How to use it
The new test quality table responds to the same filters as other coverage dashboard metrics, so you can use filters like application and environment to drill down into a specific segment of your test suite.
During a workspace review, you can review the filtered results in the test quality table to identify which tests need maintenance. For example:
- Identify flaky tests: A test with low stability and medium reliability is likely flaky and may need its locators or wait conditions updated.
- Identify broken tests: A test with high stability and low reliability is consistently failing and may reflect a genuine application issue or an outdated test flow.
- Gauge CI/CD readiness: Use the reliability metric to determine when a test is stable enough to integrate into your CI/CD pipeline.
Learn more
To learn more about the coverage dashboard, see the help docs. If you have any questions, please feel free to contact us or reach out through in-app chat.