Plan stages

Parallel vs. sequential runs; set-up and tear-down stages

Introduction to plan stages

327327

Plan stages can be used to control how tests are run within a plan. By grouping your tests into stages, you will be able to control the concurrency of the tests in each stage (parallel or sequential), the order in which tests will run, and additional settings such as browsers and DataTables.

Editing plan stages

For new plans and plans that were created prior to the introduction of stages, the default behavior is to have a single parallel stage. These settings can be changed by editing the stage or adding new stages.

20982098

After clicking the edit button you can then specify the properties of the stage in the Stage Options modal.

404404

Stage options

The following settings can be configured from the Stage Options modal:

  • Parallel: All tests in this stage will start running at the same time By default, up to 50 tests in this stage will run at the same time. Use the “Max” input to configure a different concurrency limit for the stage.

👍

Parallel test runs

In order to minimize run time and dependencies between tests, we recommend creating tests that can be run in parallel whenever possible and setting a plan stage concurrency limit to control the overall load on your test environment.

  • Sequential: tests in this stage will run one at a time in the order in which they are defined in the plan.

📘

Sequential stage execution

Stages that are configured to run tests sequentially will always run only one test at a given time. If a plan runs tests on multiple browsers, environments, and/or DataTable scenarios, only one test (on one of the configured browsers/environments/scenarios) will run at a given time.

  • Always run stage: The stage will always run, even if tests in a previous stage failed. (This property is useful for defining clean-up stages that remove test data that was generated by an earlier stage.)
  • Always run all tests in stage: If this feature is enabled, all tests in a stage run until they either pass or fail. If this feature is disabled, a failed test stops subsequent tests in the stage from running. The tests that were scheduled to run after the failed test have a skipped status.
  • Override device settings for this stage: Use a different set of browsers for this stage.
  • Override run multiplier for this stage: Use a different run multiplier for this stage.
  • Ignore data tables for this stage: Data tables will not be used with any tests in this stage.

Benefits of setting concurrency limits

Setting a concurrency limit is recommended for all plan stages, as it provides a number of benefits:

  • Most applications, particularly in pre-production environments, have limited underlying resources. A reasonable concurrency limit allows you to take advantage of speeding up your tests by running in parallel without risking overloading the underlying servers
  • Leveraging concurrency limits allows you to avoid adding excess plan stages just to limit test tests in parallel or enable stopping once you get a failure
  • When tests fail or are flaky in a parallel plan but pass individually, you can reduce the concurrency limit before resorting to running fully in sequence

Saving stage options

Once you have defined your stages and saved the plan, you will see your tests grouped by stage in the upper right of the plan page:

782782

Ordering of plan stages

The order in which stages are defined in a plan determines the order in which each stage will run. A later stage will not begin running until all previous stages have completed. This behavior is useful for modeling standard set-up / test / clean-up behavior. For example, a hypothetical plan might be define three stages like the following:

1. Set-up
Concurrency: sequential
Tests:

  • Create user
  • Add vehicle

2. Vehicle tests
Concurrency: parallel
Tests:

  • Schedule car wash
  • Order new tires
  • Run diagnostics

3. Clean-up
Concurrency: sequential
Always Run: true
Tests:

  • Remove vehicle
  • Remove user

Behavior on failure

Depending on the plan stage settings, a failing test can cause different outcomes:

  • If a test fails in a sequential stage that isn't set to always run other tests in the stage, all other tests that come after the failed test will be skipped (i.e. will not start running).
  • All tests in non-always-run stages will immediately be skipped
  • The first always-run stage (if one exists) will begin running

As an example, the following image shows what happened when a sequential test failed in the first of three stages. The third and fourth tests in the first stage were skipped, all tests in the second stage were skipped, and the tests in the third (always-run) stage were run:

23702370