Plan stages

Parallel vs. sequential runs; set-up and tear-down stages

Introduction to plan stages

Plan stages can be used to control how tests are run within a plan. By grouping your tests into stages, you will be able to control the concurrency of the tests in each stage (parallel or sequential), the order in which tests will run, and additional settings such as browsers and DataTables.

👍

Parallel test runs

In order to minimize run time and dependencies between tests, we recommend creating tests that can be run in parallel whenever possible and setting a plan stage concurrency limit to control the overall load on your test environment.

Properties of plan stages

A plan stage groups tests together and adds controls that allow you to decide how those tests should be run. The settings that can be changed in a plan stage are concurrency, always run stage, always run tests in stage, and settings to override default plan settings for browser, run multiplier, and DataTables. Here are the details on each setting:

  • Parallel: All tests in this stage will start running at the same time By default, up to 50 tests in this stage will run at the same time. Use the “Max” input to configure a different concurrency limit for the stage.
  • Sequential: tests in this stage will run one at a time in the order in which they are defined in the plan.
  • Always run stage: The stage will always run, even if tests in a previous stage failed. (This property is useful for defining clean-up stages that remove test data that was generated by an earlier stage.)
  • Always run all tests in stage: A failing test will not terminate other tests in the same stage.
  • Override device settings for this stage: Use a different set of browsers for this stage.
  • Override run multiplier for this stage: Use a different run multiplier for this stage.
  • Ignore data tables for this stage: Data tables will not be used with any tests in this stage.

📘

Sequential stage execution

  • Plans with multiple browsers configured will execute sequential stages one browser at a time. The order of execution is not configurable.
  • Tests with DataTables in a sequential stage will also be executed one scenario at a time.

Benefits of setting concurrency limits

Setting a concurrency limit is recommended for all plan stages, as it provides a number of benefits:

  • Most applications, particularly in pre-production environments, have limited underlying resources. A reasonable concurrency limit allows you to take advantage of speeding up your tests by running in parallel without risking overloading the underlying servers
  • Leveraging concurrency limits allows you to avoid adding excess plan stages just to limit test tests in parallel or enable stopping once you get a failure
  • When tests fail or are flaky in a parallel plan but pass individually, you can reduce the concurrency limit before resorting to running fully in sequence

Creating a plan with stages

For new plans and plans that were created prior to the introduction of stages, the default behavior is to have a single parallel stage. These settings can be changed by editing the stage or adding new stages.

After clicking the edit button you can then specify the properties of the stage:

Once you have defined your stages and saved the plan, you will see your tests grouped by stage in the upper right of the plan page:

Ordering of plan stages

The order in which stages are defined in a plan determines the order in which each stage will run. A later stage will not begin running until all previous stages have completed. This behavior is useful for modeling standard set-up / test / clean-up behavior. For example, a hypothetical plan might be define three stages like the following:

1. Set-up
Concurrency: sequential
Tests:

  • Create user
  • Add vehicle

2. Vehicle tests
Concurrency: parallel
Tests:

  • Schedule car wash
  • Order new tires
  • Run diagnostics

3. Clean-up
Concurrency: sequential
Always Run: true
Tests:

  • Remove vehicle
  • Remove user

Behavior on failure

Depending on the plan stage settings, a failing test can cause different outcomes:

  • If a test fails in a sequential stage that isn't set to always run other tests in the stage, all other tests that come after the failed test will be skipped (i.e. will not start running).
  • All tests in non-always-run stages will immediately be skipped
  • The first always-run stage (if one exists) will begin running

As an example, the following image shows what happened when a sequential test failed in the first of three stages. The third and fourth tests in the first stage were skipped, all tests in the second stage were skipped, and the tests in the third (always-run) stage were run:


Did this page help you?