Optimizing test execution with local, CI, and cloud runs

In mabl, you can run tests using our cloud infrastructure or using local resources - either your local machine or a container where a CI tool is installed. Deciding where to run your tests ultimately depends on your priorities, which can change throughout the development lifecycle.

This article explores how you can use mabl's different execution environments to get the most out of your test runs:

• Evaluate your testing priorities
• Build a balanced workflow

Evaluate your testing priorities

Before you can identify an approach that works for your team, consider your testing priorities. As your priorities shift during the development lifecycle, the decision of where to run your tests may change.

Value

What do you want to get out of your test runs?

While cloud runs offer the most robust capabilities, including detailed test diagnostics, parallel runs, and cross-browser validation, they also consume credits. If your credit allocation isn’t enough to cover everything that you want to validate, consider how you can conserve cloud runs for higher-impact tests:

• Use unlimited local and CI runs to catch issues earlier: Local and CI runs return a quick pass/fail result. For failing tests, follow up with cloud runs to get more details.
• Leverage test types that consume fewer credits: create plans that validate your application’s underlying API before running any associated browser or mobile tests. Only trigger the browser or mobile tests if the API tests pass.

Scale

How many tests are you running? How often are you running them?

Consider the scale of testing at different points during the development lifecycle:

• Updating and running a single test to make sure it works as expected
• Running a few smoke tests or sanity checks on every commit
• Running nightly regression suites to ensure changes don’t impact existing functionality

In any situation, you want to choose an execution environment that delivers feedback fast. Use these guidelines to inform your decisions on testing scale:

• For rapid iteration and debugging on a smaller number of tests, use local and CI runs: local and CI runs execute faster because they don’t collect test diagnostics. Since they can’t run in parallel, they are less suitable for executing larger sets of tests.
• For more robust validation on a larger number of tests, use cloud runs: with tests running in parallel in the mabl cloud, you can speed up validation for larger test suites and act on feedback faster.
• Use the mabl CLI to automate your workflow: integrate mabl CLI commands into your scripted workflows to automate test execution in the CI Runner and the mabl cloud.

Build a balanced workflow

By balancing the strengths of local, CI, and cloud runs, you can develop an effective automation strategy for testing on new features, updates, and bug fixes. The following scenario illustrates how a team might combine local, CI, and cloud runs in their development workflow to release with confidence.

Pre-commit stage: local runs

A developer builds out new search bar functionality. To make sure their changes aren’t breaking core functionality, they run tests locally.

Build stage: CI runs

During the build stage, the team wants to run tests to validate every commit to staging. The QA team configures the mabl CI runner to run smoke tests on commit. The smoke tests validate core functionality of the search bar, including input validation, error handling, and filtering.

Pre-deployment stage: cloud runs

Before releasing the new search bar to prod, the team executes the entire regression suite in a pre-production environment to ensure the search bar is ready. They configure their pipeline to create a deployment event, triggering multiple plans to run in parallel in the cloud.

Scheduled jobs: cloud runs

To ensure application stability, the team sets up a schedule to run full regression plans in the cloud on a nightly or weekly basis.