Test execution overview

mabl gives you great flexibility and control over when and where to run tests. From running a single test to validate it passes to running hundreds of regression tests in parallel across browsers, the choice is yours. You can easily implement a continuous testing strategy by running tests manually, on a schedule, and on a deployment event in your CI/CD pipeline so that every environment is covered with end-to-end tests.

Continuous testing with mabl

Continuous testing with mabl

Use cases

Here are some common use cases to help you better understand when and how to run tests:

  • Replay test changes with the mabl Trainer on your local machine to validate everything works as expected.
  • Trigger an ad-hoc test run in the mabl cloud against your staging environment to troubleshoot things with all available diagnostics data.
  • Use the mabl CLI to run multiple tests in headless mode against a local development branch to validate that your feature changes didn’t break something.
  • Run mabl end-to-end tests in headless mode as part of your JavaScript testing framework to catch issues as early in the development process as possible (available as private beta; request access)
  • Run test changes saved to a mabl branch to test code that hasn’t yet been merged to a master or release branch
  • Use the mabl Runner to run mabl tests in headless mode inside your CI/CD environment to perform end-to-end tests on every code change, fast (available as private beta; request access).
  • Trigger smoke and regression test runs in the mabl cloud on every deployment to staging and production environments in your CI/CD pipeline
  • Test every code commit to a pull request (PRs) and see the test results directly inside the GitHub or BitBucket PR.
  • Schedule plans to run all regression tests overnight against your staging environment across multiple browsers, including Internet Explorer 11 and Safari.
  • Schedule a plan with key user journeys to monitor your production environment every few hours and get an alert when there is an issue with the user experience

Test execution

You can run mabl tests in the following ways against any of your public or private test environments:

  • Local - tests run on your own machine using the mabl Trainer or the mabl Runner, which is part of the mabl CLI
  • Self-hosted - tests run inside your own CI/CD containers (servers) using the mabl Runner (available as private beta; request access)
  • mabl cloud - tests run inside mabl’s highly secure and scalable infrastructure without limit on parallel runs
mabl test execution environments

mabl test execution environments

Single test runs

As you create tests with the mabl Trainer, you can quickly replay the trained steps on your computer to validate that everything works as expected before saving your changes. Once the test is created, you can head over to the test details page in the mabl app to manually trigger an ad-hoc test run in the mabl cloud using the Run test button.

Multiple test runs

To run multiple tests as a group you should add them to a plan or tag them with a label. You would use test labels to do feature-based testing by running a small number of self-contained tests. On the other hand, plans give you the ability to specify test executions (e.g. parallel vs sequential, setup vs teardown), schedule runs, pass variables between tests, and more. You can also tag plans with labels to trigger multiple plan runs on a deployment event, for example.

Plan runs

You can use plans as a grouping mechanism to run tests in headless mode locally or in your CI/CD environment. This can be achieved using the --from-plan option of the mabl tests run-alpha command in the mabl CLI. However, plan features such as parallel runs, stages, and variable sharing between tests are not supported in the mabl Runner at the moment.

Here is how the different mabl test execution environments compare:

Local
CI/CD*
mabl cloud

Feedback speed: single test

Fastest

Faster

Fast

Feedback speed: multiple tests

Fast

Faster

Fastest

Parallel runs

No

No

Yes (unlimited)

Scheduling

DIY (bash scripts)

DIY (CI Tool)

Yes

Results reporting in mabl

No

No

Yes

Test diagnostics data

Pass/Fail only

Pass/Fail only

Yes

Cross-browser support

Chrome only

Chrome only

Yes

Plan runs support (stages, etc.)

Only as a grouping mechanism

Only as a grouping mechanism

Yes

Insights (auto-heal, visual, JavaScript, etc.)

No

No

Yes

Needs secure tunnel (mabl Link)

No

No

Yes

Counts towards monthly test runs

No

To be announced

Yes

*Running mabl tests inside your own CI/CD environment is available as a private beta and you can request access here.

Updated 27 days ago


Test execution overview


Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.