Test execution overview

How to run tests in mabl

Mabl gives you great flexibility and control over when and where to run tests. From running a single test locally to validate it passes to running hundreds of regression tests in parallel across browsers, the choice is yours. You can also implement a continuous testing strategy by running plans manually, on a schedule, and on a deployment event in your CI/CD pipeline so that every environment is covered with end-to-end tests.

1637

Continuous testing with mabl

Here are some common use cases to help you better understand when to run tests:

  1. Run individual tests for quick feedback and validation
  2. Run multiple tests in a plan run or deployment event to validate that a feature works as expected before introducing new code into your application
  3. Integrate tests into your CI/CD workflow to catch issues as early in the development process as possible
  4. Set up smoke and regression tests to run on a schedule to ensure everything continues to work as expected
  5. Run tests from the mabl CLI inside your local environment to validate your changes before adding them to your application.

You can run mabl tests locally, in the cloud, or in a self-hosted CI/CD environment against any of your public or private test environments.

Local runs

Local runs execute locally to your machine and do not count towards your monthly quota of allotted tests.

👍

Quick feedback

Local test runs work best as a diagnostic tool to get quick feedback on a single test.

Cloud runs

Tests executed in the cloud run on mabl’s highly secure and scalable infrastructure. You can run individual tests in the cloud or run tests as part of a plan.

Plan runs

Plan runs give you the ability to configure even more settings for test execution, including:

Concurrency limits

The following table shows the maximum number of tests that can run in parallel at a given time for different types of tests:

TypeTrial userCustomer
Browser test101000
API test251000

If you trigger more than the maximum parallel test run limit, the remaining tests have an on hold status until the other test runs finish. If you need higher concurrency, reach out to the mabl Support Team.

Maximum run times

The maximum run times for cloud runs are as follows:

  • Browser tests: six hours
  • API tests: 30 minutes

📘

Browser tests in the cloud

Keep in mind the following for browser tests run in the cloud:

  • Every cloud run uses a new instance, so mabl doesn’t cache your test data, including user sessions, between runs.
  • The timezone in cloud runs is UTC.

Self-hosted - CI Runner

Tests run inside your own CI/CD containers (servers) using the mabl CI Runner.

Comparison of test execution environments

A full comparison of the different mabl test execution environments appears in the following table:

LocalCI Runner (Self-hosted in CI/CD)mabl cloud
Feedback speed: single testFastestFasterFast
Feedback speed: multiple testsFastFasterFastest
Parallel runsNoNoYes
SchedulingDIY (bash scripts)DIY (CI Tool)Yes
Results reporting in mablNoNoYes
Test diagnostics data (for example screenshots, HAR logs, console logs)Pass/Fail onlyPass/Fail onlyYes
Multibrowser supportChrome onlyChrome onlyYes
Plan runs supportOnly as a grouping mechanismOnly as a grouping mechanismYes
InsightsNoNoYes
Needs secure tunnel (mabl Link)NoNoYes
Counts towards monthly test runsNoNoYes
Test runtime limitNoNoBrowser tests: 6hr
API tests: 30 min