Plan stages can be used to control how tests are run within a plan. By grouping your tests into stages, you can:
- Configure tests to run in parallel or sequentially
- Manage behavior on failure
- Override default settings for browsers, the run multiplier, and DataTables
This guide explains the different settings that can be applied to plan stages: concurrency, behavior on failure, and override settings.
Best practices for organizing plans
If you're seeking advice for how to organize tests into plans, check out the guide Organizing tests into plans.
To get started, take the following steps to open the Stage Options modal:
- Open the Plan Details page
- Click on the pencil icon to open the Plan Edit page.
- Scroll down to the Tests section to find the plan stages.
- Click on the pencil icon next to a stage to open the Stage Options modal.
Tests in a stage can run in parallel or sequentially.
In a parallel stage, all tests start running at the same time. Running tests in parallel helps minimize run time and dependencies between tests.
When a stage is configured to run tests in parallel, up to 50 tests run at the same time by default. Use the "Max" input to configure a different concurrency limit.
Setting a concurrency limit is recommended for all plan stages, as it provides a number of benefits:
- Most applications, particularly in pre-production environments, have limited underlying resources. A reasonable concurrency limit allows you to take advantage of speeding up your tests by running in parallel without risking overloading the underlying servers
- Leveraging concurrency limits allows you to avoid adding excess plan stages just to limit test tests in parallel or enable stopping once you get a failure
- When tests fail or are flaky in a parallel plan but pass individually, you can reduce the concurrency limit before resorting to running fully in sequence
Tests in a sequential stage run one at a time in the order in which they are defined in the plan.
Behavior of sequential stage execution
If a plan runs tests on multiple browsers, environments, and/or DataTable scenarios, only one test (on one of the configured browsers/environments/scenarios) runs at a given time.
Depending on the plan stage settings, a failing test can cause different outcomes. The Stage Options modal has two settings to manage behavior on failure: always run all tests in stage and always run stage.
If this setting is enabled, all tests in a stage run until they either pass or fail.
If this setting is turned off, a failed test stops subsequent tests in the stage from running. The tests that were scheduled to run after the failed test have a skipped status.
If this setting is enabled, the stage always runs even if tests in a previous stage failed. This setting can be useful for defining clean-up stages that remove test data generated in a previous stage.
If this setting is turned off, tests in this stage only run if all tests in previous stages passed. If a test in a previous stage fails, the tests in this stage have a skipped status.
Use a different set of browsers for this stage.
Use a different run multiplier for this stage.
If the "Ignore data tables for this stage" setting is enabled, tests that are associated with DataTables only run the first scenario of the first DataTable associated with the test.
Updated 7 months ago