mabl makes it easy and fast to run reliable end-to-end tests at every stage of your CI/CD pipeline. Continuous testing across CI/CD environments gives peace of mind to software developers, QA engineers, product managers, and other business stakeholders that issues are proactively resolved.
There are two ways to run mabl tests as part of your DevTestOps workflow — after a deployment to a test environment or even earlier in the development process, after a code build. This article provides an overview of those two options, including considerations around testing pull (merge) requests, and references additional how-to resources.
Test on deployment
Integrating mabl tests in your deployment process helps you test the end-to-end user experience in multiple browsers as soon as code is deployed to a hosting environment within your CI/CD pipeline. All mabl tests can run across environments by design, so you can easily run the same tests across multiple environments — from ephemeral preview environments for testing pull/merge requests to persistent staging environments, and even production! Some bugs may only reveal themselves in a production environment, so we shouldn't be afraid to also test there.
Example of a CI/CD pipeline with 3 environments.
You can integrate mabl with your CI/CD pipeline to run tests on deployment using one of the following options, in order of ease of use:
- Existing CI/CD integration - GitHub Actions, Bamboo, and Jenkins, among others.
- mabl CLI - integrate with any CI tools that can install and run Node.js packages.
- mabl APIs - integrate with any CI tools where you cannot use the mabl CLI or need more control.
Example of a report for all recent deployment test results.
Use one of the options, such as the mabl CLI, to create a deployment event in mabl that tells mabl what test plans to run, where to run them (e.g. environment, browsers), and what other configuration options to apply (e.g. credentials, HTTP headers).
Test organization considerations
It is important that you organize your tests in one or more plans that you want to run on deployment. You can use plan labels to control which plans to run. At present, it is not possible to use test labels to run a group of tests for a deployment event in mabl.
We recommend that you run tests in the mabl cloud when you are deploying to an environment so that you can benefit from quick feedback when you run hundreds of tests in parallel as well as all the rich diagnostics data (e.g. DOM snapshots, network activity) and insights (e.g. visual change) that mabl generates when running tests in the mabl cloud.
Example of test plan results for a deployment event. Detailed logs for individual test runs are a click awayl
Test pull requests
Many software development teams run unit tests, static analysis, and various other code checks as part of their pull request (PR) review process. Those checks are very valuable because they help catch issues early on in the development process and also make the code review process faster by providing PR reviewers with insight into the existence or the lack there of code quality regressions.
Pull request vs merge request
Pull request is a term used by source control management (SCM) tools like GitHub and BitBucket to describe the process of notifying your team that a code change has been pushed to a git branch and is ready for review. The word “pull” comes from the first git command (git pull) that a git user needs to perform in order to get a local copy of the source code for the branch that is to be reviewed and merged. If everything looks good, then the git user will run a git merge command to merge the code change into the main branch. Thus, other SCM tools like GitLab use the term merge request instead of pull request but essentially they mean the same thing. The mabl help docs use only one of those terms - pull request - for consistency.
Historically, teams have avoided adding end-to-end UI tests as part of the PR checks because those tests were flaky, brittle, and very slow to run. However, this is no longer the case thanks to mabl’s dynamic waits, auto-healing, parallel execution, headless mode, and more. Performing end-to-end testing on the UI and/or API layers at the PR stages gives your team even higher confidence in approving the code changes and helps you accelerate your continuous delivery process.
mabl test results shown as a GitHub PR check.
You can start testing your PRs with mabl by using the integration with GitHub Checks or BitBucket Code Insights. Those integrations assume that your team deploys the pushed PR code to a test environment, usually ephemeral one, that you can test from the mabl cloud. You can establish a secure tunnel with mabl Link, if needed.
You can also take a different approach. If a test environment for the PRs is not available or you simply do not want to run tests in the mabl cloud for every PR, you can configure the code repository to use the mabl runner and run tests in headless mode during the build process as described in the next section.
Test on build
In addition to running tests locally and in the mabl cloud, you can use the mabl Runner to run end-to-end tests in headless mode inside your CI environments during a code build. This enables you to shift-left and get feedback from end-to-end tests earlier in your development lifecycle without having to deploy your application to a test environment. The tests and the application under test will both run inside the CI container or virtual machine (VM).
The primary distinction with the mabl Runner is that these tests run inside of your CI/CD infrastructure. Unlike cloud runs, mabl Runner test runs are limited to Chrome, they do not generate auto-heals or rich diagnostic artifacts (e.g. screenshots, HAR files, etc.), and the results are not stored in the mabl app. On the other hand, CI runs are efficient and generally complete in a fraction of the time that it takes to execute tests in the cloud. See the Test execution overview page for a more detailed comparison between the mabl Runner and cloud runs.
To get started with the mabl Runner, please see running tests inside the CI environment