Test execution overview

When and how to run tests in mabl

Mabl gives you great flexibility and control over when and where to run tests. From running a single test locally to validate it passes to running hundreds of regression tests in parallel across browsers, the choice is yours. You can also implement a continuous testing strategy by running plans manually, on a schedule, and on a deployment event in your CI/CD pipeline so that every environment is covered with end-to-end tests.

16371637

Continuous testing with mabl

When to run tests

Here are some common use cases to help you better understand when to run tests:

  1. Run individual tests for quick feedback and validation
  2. Run multiple tests in a plan run or deployment event to validate that a feature works as expected before introducing new code into your application
  3. Integrate tests into your CI/CD workflow to catch issues as early in the development process as possible
  4. Set up smoke and regression tests to run on a schedule to ensure everything continues to work as expected
  5. Run tests from the mabl CLI inside your local environment to validate your changes before adding them to your application.

How to run tests

You can run mabl tests locally, in the cloud, or in a self-hosted CI/CD environment against any of your public or private test environments.

Local runs

Local runs execute locally to your machine and do not count towards your monthly quota of allotted tests.

👍

Quick feedback

Local test runs work best as a diagnostic tool to get quick feedback on a single test.

Cloud runs

Tests executed in the cloud run on mabl’s highly secure and scalable infrastructure.

You can run individual tests in the cloud or run tests as part of a plan. Plan runs give you the ability to configure even more settings for test execution, including:

Concurrency limits
The following table shows the maximum number of tests that can run in parallel at a given time for different types of tests:

Type

Trial user

Customer

Browser test - Chrome, Firefox, and Edge

10

1000

Browser test - Safari

10

15

Browser test - IE

10

50

API test

25

1000

If you trigger more than the maximum parallel test run limit, the remaining tests have an on hold status until the other test runs finish. If you need higher concurrency, reach out to the mabl Support Team.

📘

Maximum run times

The maximum run times for cloud runs are as follows:

  • Chrome, Firefox, Internet Explorer, Edge: six hours
  • Safari: one hour
  • API tests: 30 minutes

Self-hosted (CI Runner)

Tests run inside your own CI/CD containers (servers) using the mabl CI Runner.

A full comparison of the different mabl test execution environments appears in the following table:

Local

CI Runner (Self-hosted in CI/CD)

mabl cloud

Feedback speed: single test

Fastest

Faster

Fast

Feedback speed: multiple tests

Fast

Faster

Fastest

Parallel runs

No

No

Yes

Scheduling

DIY (bash scripts)

DIY (CI Tool)

Yes

Results reporting in mabl

No

No

Yes

Test diagnostics data (for example screenshots, HAR logs, console logs)

Pass/Fail only

Pass/Fail only

Yes

Multibrowser support

Chrome only

Chrome only

Yes

Plan runs support

Only as a grouping mechanism

Only as a grouping mechanism

Yes

Insights

No

No

Yes

Needs secure tunnel (mabl Link)

No

No

Yes

Counts towards monthly test runs

No

No

Yes

Test runtime limit

No

No

Chrome, Firefox, Edge: 6hr
Safari: 1hr
API Test: 30 min