mabl gives you great flexibility and control over when and where to run tests. From running a single test locally to validate it passes to running hundreds of regression tests in parallel across browsers, the choice is yours. This article explains the different test execution environments available in mabl: local, self-hosted - CI Runner, and cloud.
To learn more about how you can leverage different execution environments in a continuous testing workflow, see our article on optimizing test execution with local, CI, and cloud runs.
Comparison of test execution environments
A full comparison of the different mabl test execution environments appears in the following table:
| Local | CI Runner (Self-hosted in CI/CD) | mabl cloud | |
| Results reporting in mabl | No | No | Yes |
| Test run artifacts, such as screenshots, HAR logs, and console logs | Pass/fail only | Pass/fail only | Yes |
| Parallel runs | No | No | Yes |
| Plan runs support | Only as a grouping mechanism | Only as a grouping mechanism | Yes |
| Scheduling | DIY (bash scripts) | DIY (CI Tool) | Yes |
| Cross-browser validation | No - Chrome only | No - Chrome only | Yes |
| Counts towards allotted credits | No | No | Yes |
| Feedback speed: single test | Fastest | Faster | Fast |
| Feedback speed: multiple tests | Fast | Faster | Fastest |
| Requires Link or allowlisting IP addresses to access private environments | No | No | Yes |
| Test runtime limit | No | No |
Browser - 6 hr API - 30 min Mobile - 1 hr |
| Performance tests | No | No | Yes |
| GenAI Assertions | CLI - No | No | Yes |
| Desktop - Yes | |||
| Insights | No | No | Yes |
Local runs
mabl supports triggering tests to run on your local machine using the Desktop App or the mabl CLI. Local runs work best as a quick diagnostic tool to get feedback when creating tests or debugging issues.
For a local run, mabl execution logs and results are displayed in a separate test output window:
Logs for a headless local run
Unlike cloud runs, the logs for local runs do not persist in the mabl app. Local runs do not capture test run artifacts, such as DOM snapshots and screenshots.
Limitations of local runs
Self-hosted - CI Runner
With the mabl CLI, you can set up the CI Runner to run tests inside your own CI/CD containers (servers). CI runs are ideal for lightweight, targeted tests, such as smoke or sanity tests, where you’re looking for a quick pass/fail result before deployment to shared environments.
Using CLI commands directly in your CI/CD pipeline, you can configure environments and leverage test labels to automate validation of tests that target core functionality.
Limitations of CI runs
- Plan executions are not supported in the CI Runner.
- No support for parallel runs.
- Browser runs are only supported for Chrome.
- No support for performance tests.
Cloud runs
Tests executed in the cloud run on mabl’s highly secure and scalable infrastructure. The following types of executions are supported for cloud runs:
- Ad hoc runs
- Plan runs - run plans in parallel or sequentially, including support for scheduled runs and sharing variables across tests
- Deployment events - run multiple plans in a deployment event using the mabl CLI, mabl API, or an existing CI/CD integration
Every cloud run uses a new instance, so mabl doesn’t cache your test data, including user sessions, between runs.
Cloud runs collect detailed test run artifacts in the Results section of the app. You can use the reporting API to integrate cloud run results into your team’s preferred reporting tool. For more information on cloud run artifacts by test type, check out the following articles:
- Browser test output
- Reviewing mobile tests
- Reviewing API tests
- Understanding performance test output
- Visual test output
Concurrency limits
The following table shows the maximum number of tests that can run in the cloud in parallel at a given time:
| Type | Trial user | Customer |
| Browser test | 10 runs | 1000 runs |
| Mobile test |
5 Android runs 3 iOS runs |
10 Android runs 5 iOS runs |
| API test | 25 runs | 1000 runs |
If you trigger more than the maximum parallel test run limit, the remaining tests have an “on hold” status until the other test runs finish. If you need higher concurrency, reach out to the mabl Support Team.
Maximum run times
The maximum run times for cloud runs are as follows:
- Browser tests: six hours
- API tests: 30 minutes
- Mobile tests: one hour