When it comes to test results, performance test runs work a bit differently than functional test runs. This article serves as a reference for key concepts on the performance test output page so you can make sense of the results:
If you’re looking for guidance on how to filter and interpret performance test results, see the article on reviewing performance tests.
Status
Keep in mind the following points when reviewing performance test status:
Pass/Fail
Performance tests fail if the threshold for at least one failure criterion is met. If a performance test has no failure criteria, it will always pass.
For example, a performance test with no failure criteria and a 100% functional test failure rate will still be marked as passing.
Stopped performance tests
If you stop a performance test, it takes up to one minute for all tests to stop running. Stopped performance tests consume the number of billable VUH up until the test stopped
Duration
Duration indicates the total amount of time it took for the performance test to run to completion. This time includes the duration configured for the performance test plus the time it takes to provision all the resources required to run the test and cleanup time at the end.
For example, if you trigger a performance test that is configured to run for 10 minutes, the load will run for exactly 10 minutes, but the total duration on the test output page might be 11 minutes or 15 minutes, depending on how long it takes to spin up all the runners required for the test.
The total amount of time required to provision resources for the performance test run varies from one run to the next.
Usage
Performance test usage is measured in credits. To determine the number of credits consumed for a performance test run, multiply the number of billable VUH shown on the test output page by 4. Billable VUH represents the total number of billable virtual user hours (VUH), billed in 15 minutes.
For example, a performance test that shows 3 Hours as the billable VUH consumes a total of 12 credits.
Functional test runs
Performance test output includes the number of functional test runs that passed and failed during the test.
If the functional tests running in a performance test take longer than the configured duration of a performance test, the total count of passed and failed functional test runs will be 0.
Performance test metrics
Performance test metrics include metrics for both browser and API tests. For a breakdown of each metric, see the article on measuring system performance.