Release coverage

Filtering and interpreting data in the Release Coverage page

Whether you're looking to understand the readiness of a new release or new version of your app, or you're just curious to check in on a key set of tests, the release coverage page in mabl allows you to quickly view the current status of your testing in one centralized place.

Visit the coverage page now to explore.

13661366

The Release Coverage page

This guide provides an overview on how to filter and interpret metrics on the Release Coverage page.

Filtering data

The filters are the starting point for using the Release Coverage page. When you set and remove filters, the data in the Release Coverage page updates to count only the tests that match your filters.

10661066

Release Coverage filters

By setting the filters to your definition of a release, such as tests with a specific feature label or all tests in the dev environment in the past two weeks, you can use the metrics on the Release Coverage page to monitor your key tests and user flows.

Metrics at a glace

Use the high-level metrics cards to get a quick understanding of your test suite, as defined by the filters.

12171217

Latest pass rate

The "Latest pass rate" tells you the most recent status of the active tests in your defined test suite. This metric can help you understand the health of a release. For example, if you ran a suite of tests for an upcoming release, you can use the latest pass rate to understand what percentage of your test suite passed as of the most recent run.

Cumulative tests run

The "Cumulative tests run" chart shows how many unique tests ran out of all the active tests that match the filters you set within the specific date range. This metric gives a quick idea of what percentage of your test suite you’re running for a release, as defined by your filter. To find out which specific tests haven't run, use the Test Status table.

Performance

If you're on an Enterprise plan, you can use the performance card to identify potential performance regressions at the release level. The performance card shows the Browser and API tests that had the greatest slowdown in performance in the underlying pages and endpoints accessed by those tests. You can also see an overall metric for the increase/decrease in average app load time and average API response time, averaged across all tests run for the release, as defined by your filters.

Click on the test or percent change to view more details on that specific test's performance.

📘

Measuring performance

Performance data is calculated from tests run as part of a plan. Data from ad-hoc runs are not included in performance metrics.

Quality metrics

The Quality Metrics tab gives more detail on the test run history, average app load time, and categorized failures for the filtered group of tests.

Test run history

The "test run history" chart shows the run history across the date range that you set in the filters.

11801180

Average app load time

Enterprise users have access to the average app load time for your application as filtered on the Release Coverage page. This data is collected automatically from the Google Lighthouse's "Speed Index" metric for every mabl test running on Chrome and Edge in the cloud, so long as they're not run ad hoc.

This chart helps you identify larger performance trends, both improvements and regressions, across all of your testing.

869869

Categorized failures

If you add failure reasons to failed test runs, you can use this chart to categorize test failures.

11741174

Test status

The Test Status table displays results for specific tests with a handy set of filters.

📘

Note

The Avg. performance and Performance change column headers only appear in workspaces on an Enterprise plan.

12051205

The Test Status table

Using the filters in the Test Status table, you can:

  1. Find out which tests haven't run.
  2. Uncover broken tests
  3. Identify long-running tests
  4. Identify performance change for individual tests

Find out which tests haven't run

Click on the Latest status column header to find out which tests have the Not run indicator. You can either visit these tests directly to kick off an ad hoc run, or view them to add them to an existing plan to close up that gap in testing coverage.

Uncover broken or unused tests

Click on the Pass rate column header to identify tests that have run and have a pass rate of 0%. Additionally, click on the # of runs column header to find tests that have 0 runs. You can click the link to access these tests to fix them or turn them off to exclude them from the page entirely.

Identify long-running tests

If you want to optimize your test suite, click on the Avg test run time column header to identify long-running tests. See our guide on optimizing test performance for suggestions on how to reduce the overall run time of your tests.

Identify performance change for individual tests

If you want to view performance change data beyond the top three tests that appear in the performance card, click on the Performance change column header. This column header only appears in Enterprise workspaces. When you click on the percent change, you can view the performance page for that particular test.


Related resources