Change validation tells your pipeline whether a code change can move forward at each stage of your workflow, from a pull request through the merge queue to a deployed commit. At each stage, mabl runs the suites you choose, plus the tests the change impacts, found by test impact analysis (TIA). Your pipeline then passes or blocks the change based on the results. This article explains what mabl runs at each stage and how it reaches a result. To set up change validation, see Run change validation from your own CI.
Early access
Change validation is in early access for accounts with generative AI features. Pricing and packaging are subject to change.
The policy file
A policy file in your repository, .mabl/policy.yaml, defines the stages of your workflow and what mabl checks at each one. Stage names are your own, such as pull_request or merge_queue, and mabl doesn't attach any meaning to them. Your pipeline decides when each stage runs and asks mabl for it by name.
Each stage lists one or both kinds of checks:
- Suite checks run the plans you name, such as your smoke tests, every time the stage runs.
- Impacted checks run the tests the change affects, within the rules your policy sets.
For example, this policy runs the smoke suite and the impacted tests on every pull request, and only the smoke suite has to pass:
version: 1
suites:
smoke:
plan_labels: [smoke]
applications:
web:
mabl_application: '[application-id]'
target:
environment: '[environment-id]'
stages:
pull_request:
checks:
- kind: suite
suite: smoke
must_pass: true
- kind: impacted
The policy that applies to a pull request is the one on the branch the pull request targets, not the one in the pull request itself, so an author can't loosen the rules in their own change. Your pipeline reads the file from that branch and sends it each time it asks for a stage. When you edit the policy, the edit applies to pull requests once it merges. A deployed commit uses the policy file as it is at that commit.
To add a policy file, see Set up a mabl policy file. For every setting, see the policy file reference on GitHub.
Running a stage
When your pipeline asks for a stage that has an impacted check, mabl first finds the impacted tests. Then it runs those tests and the stage's suites together, in one deployment event for each application, against the environment your policy names. A test that's both impacted and part of a suite runs once.
Your pipeline reads each stage's status from mabl and reports it, for example as a check on the pull request. The tests run as a deployment event, so their results also appear in mabl like those of any other deployment.
Each stage keeps only its latest result. When a pull request gets a new commit and your pipeline asks for the stage again, the result for the earlier commit stops counting. Stages don't overwrite each other, so the merge queue stage's result doesn't replace the pull request stage's result. For a deployed commit, asking for a stage again runs it again, and the newest run is the stage's result.
Running impacted tests
Unlike the mabl test impact skill, which reads your code changes, change validation works from a summary of the change: the pull request's title, its description, and the list of files it changes. mabl doesn't read your source code. When a stage uses the pull request's mabl branch, the analysis also considers tests created or edited on that branch.
Because impacted tests run automatically, mabl filters them before any of them run. It applies these filters in order:
- Your policy's rules. Only enabled browser tests run. Your policy can exclude tests by label, limit them to certain suites or labels, or skip tests whose median run time is too long.
- Trusted labels. A test with one of the labels your policy trusts runs without the read in step 3, unless the stage uses a mabl branch that has its own version of the test.
- A read of each remaining test. mabl reads the test's steps, including its flows and JavaScript snippets, and runs the test only if no step writes data.
If mabl isn't sure a test is safe to run, it leaves the test out. A test that creates a record and then deletes it counts as writing data, and so does a test mabl can't read in full.
If an analysis finds impacted tests but few of them run, add a trusted label to the tests you're comfortable running on any pull request without mabl reading them first, such as tests that only read data or that reliably clean up everything they create. Leave the label off tests that change data other tests or teammates depend on. See Trusted labels.
Blocking a change
By default, a failed check is only a warning. It blocks a change only when your policy marks the check must_pass: true. Whether a check passes depends only on its test results, so the same results always give the same outcome.
- If a check that must pass fails, the change is blocked.
- If mabl can't judge a check that must pass, or can't judge the stage at all, the change needs attention.
- If every check that must pass has passed or been waived, and nothing is still running, the change is complete.
To let one pull request through a check that must pass, give the pull request the waiver label your policy names for that check.
Two rules keep a pass meaningful. An impacted check with nothing to run passes only if the analysis succeeded, and its result says how many tests the analysis found and why they didn't run. If the policy file is missing, invalid, or doesn't define the stage, the whole stage fails.
Automatic retries count, so a test that passes on retry counts as passed. A rerun started by hand doesn't change the result. To recover from a flaky failure, push a new commit or queue the pull request again. For a deployed commit, ask for the stage again.
For each status a check can have, see the mabl API reference.
Stages in common workflows
Because you name the stages, one policy file can fit the way your team builds and deploys. Here are three common layouts. Each example shows part of one application's entry.
A preview for each pull request. Your CI deploys each commit to its own preview, then asks for a stage. mabl runs the impacted tests against the preview, so the results reflect the change before it merges.
target:
url: 'https://pr-{sha7}.preview.example.com'
environment: '[environment-id]'
stages:
pull_request:
checks:
- kind: impacted
A merge queue. A second stage runs when a pull request enters the merge queue. Its checks must pass, so they hold the merge, while the pull request stage keeps reporting warnings.
stages:
pull_request:
checks:
- kind: impacted
merge_queue:
checks:
- kind: suite
suite: smoke
must_pass: true
- kind: impacted
must_pass: true
A shared environment. Without previews, a change isn't deployed anywhere until it merges. The pull request stage lists the impacted tests for the reviewer without running them (run: false). After the merge deploys to staging, your pipeline asks for a second stage, and mabl runs the smoke suite and the impacted tests there. The staging stage belongs to the deployed commit (scope: revision), so the pull request never waits on it. Its checks must pass, so its result can gate your next promotion.
target:
environment: '[environment-id]'
stages:
pull_request:
checks:
- kind: impacted
run: false
staging:
scope: revision
checks:
- kind: suite
suite: smoke
must_pass: true
- kind: impacted
must_pass: true
Limits
- Each stage runs at most 50 impacted tests.
- A pull request or deployed commit can have at most 10 stages.
- A pull request gets at most 50 analyses over its life. After that, mabl starts no more analyses until you reset validation for the pull request.
- mabl can't tell whether a pull request comes from a fork, so your pipeline must skip pull requests from forks.
For the full list, see Limits when a stage runs.