Performance test metrics offer a variety of ways to measure your application's performance. When combined with failure criteria, you can set thresholds indicating the minimum acceptable performance and fail tests that don't meet your team's requirements. That way, as new changes are introduced to your application, you can use performance test output to monitor for performance regressions and ensure that everything is working as expected.
This article outlines the types of performance testing metrics available for mabl performance tests.
Functional test failure rate
Functional test failure rate represents the percentage of functional tests with a "failed" status. A "failed" status indicates that assertions in the underlying functional test were violated.
You may set functional test failure rate as a failure criterion for browser and API tests.
Setting a functional test failure rate
Browser-specific performance metrics include Core Web Vitals and step duration. You can create failure criteria for these metrics for all steps or for specific steps in a browser test.
- All steps: the results from all steps are combined together into a single score for that metric. Use failure criteria for all steps to monitor the baseline performance for your application as a whole. This works best when all the steps are expected to have similar performance for the chosen metric.
- Specific steps: set failure criteria for specific steps when you want to keep a closer eye on critical areas of your application. For example, you can set failure criteria to monitor navigation steps, such as clicking links and buttons.
Setting failure criteria for specific steps in a test
Core Web Vitals
Core Web Vitals are a set of metrics defined by Google that measure different aspects of the user experience. By improving your application's Core Web Vitals, you can improve your application's SEO and its overall perceived responsiveness.
You can measure your application's performance using the following Core Web Vitals:
|First contentful paint (FCP)
|The time it takes for the browser to render the first DOM element on the page
|Largest contentful paint (LCP)
|The time it takes for the browser to render the largest image or text block on the page. This metric is considered the best approximation of a page's perceived performance.
|Cumulative layout shift (CLS)
|A measurement of how much a page shifts its contents while loading. On a page with a high cumulative layout shift, elements shift around repeatedly while they load, making it difficult for users to interact with the page.
|Time to first byte (TTFB)
|The time it takes the browser to receive the first byte of the response after making a request to the server.
|DOM content loaded
|The time it takes for the DOM to be completely loaded and parsed
Thresholds indicate "moderate" and "poor" performance based on Google's recommendation.
Step duration measures the time it takes mabl to execute a step.
If your application is a single-page application (SPA), we recommend using step duration to measure the performance of navigation steps in your application. Chrome cannot collect Core Web Vitals on subsequent navigations steps in an SPA. Google is aware of the limitations of Core Web Vitals for SPAs and have been exploring soft navigation as a solution.
API-specific performance metrics include response time and HTTP error rate.
Response time is the time it takes for the server to process the request and return a response.
In addition to averages, you can set failure criteria for API response time to fail according to percentiles. Percentiles tell you which response times fall below a given value. For example, if the API response time for the 95th percentile is 500ms, 95% of the API response times were lower than 500ms.
Percentiles are less sensitive to outliers than averages. Using percentiles to measure API response time can help you determine whether most of your users are getting good performance and give a more complete understanding of your system’s response to load.
HTTP error rate
HTTP error rate is the percentage of API requests with error responses.
There can be some overlap between HTTP error rate and functional test failure fate since API tests have default assertions to verify that requests return a 200 response.