Skip to main content

Test impact analysis in CI/CD

Molly Exten
  • Updated

Surface coverage gaps and affected tests before shipping code. By adding test impact analysis (TIA) to your CI pipeline, your team gets automated PR comments highlighting these insights, all without risking broken builds or blocking merges. You can also configure your pipeline to run the impacted tests for immediate feedback.

Follow the steps below to integrate TIA into your CI workflow.

Test impact analysis results posted as a pull request comment, with the impacted tests that ran

Want to try TIA locally?

If you work with an AI coding agent, the mabl test impact skill runs the same analysis from your editor, describes the change for you, and runs the tests it selects.

Prerequisites

Before you add test impact analysis to a pipeline, gather the following:

  • A mabl API key. We recommend a "Command Line Interface" API key. See managing API keys.
  • The ID of the application you want to analyze. See mabl resource IDs.
  • The mabl CLI installed in your pipeline. Install it globally with npm install -g @mablhq/mabl-cli, as the GitHub Actions example below does, or as a dev dependency with npm install --save-dev @mablhq/mabl-cli. With a dev dependency, run commands as npx @mablhq/mabl-cli instead of mabl.

Add test impact analysis to your CI pipeline

In any CI job that can run the mabl CLI, authenticate with your mabl API key, then run mabl tests impact with a description of the change:

mabl auth activate-key "$MABL_API_KEY"

mabl tests impact \
  --application-id {application-id} \
  --change-description-file change.md \
  --output markdown

The command waits for the analysis to finish, which takes a few minutes.

  • Passing change descriptions - To keep multi-line text out of your shell quoting, use the --change-description-file option. To pass it inline, use --change-description instead.
  • Output formats - use --output markdown for a report you can post as is, or --output json when a later step reads the result, such as a step that runs the impacted tests.

For a complete list of options, see mabl tests impact in the mabl CLI command reference.

Write the change description

Unlike the mabl test impact skill, which reads your code changes, TIA in CI can't see the diff. It only knows what the change description tells it, which is typically the pull request title and body. TIA matches that text against the content of your tests, such as step descriptions, assertions, and page names. The closer your description is to the way your tests describe your product, the more of the right tests it finds.

To get the most out of TIA in CI, adhere to the following guidelines:

  • Start with the pull request. A title and description written for a human reviewer are already most of a good change description. The GitHub Actions example below uses the PR title and body directly.
  • Describe the change in product vocabulary, not implementation detail. A description like "Memoized the validation hook in FieldGroup.tsx" won't retrieve anything because mabl tests don't describe files or classes. A better description would be "Validation now runs on each keystroke instead of on blur, so error text appears while the user is still typing."
  • For shared code, name every user-facing area the change reaches. For example, if a validator shows up at checkout and wherever a customer edits a saved address, call that out in the description.

A small map from code paths to product areas, kept in your pipeline, recovers the product vocabulary that a diff alone doesn't have:

apps/web/src/checkout/**  = checkout and payment
apps/web/src/account/**   = account settings and saved payment methods
apps/web/src/search/**    = product search and filtering
packages/forms/**         = form validation

A change that touches packages/forms/** and apps/web/src/checkout/** then adds a line to the description such as: "Product areas touched: form validation, checkout and payment."

Exit codes

The command exits 0 whenever an analysis comes back, including when the analysis finds no impacted tests at all. It exits non-zero only when the call itself fails: an API key it can't authenticate, an application ID it can't resolve, a timeout, or an account without generative AI features. If the command fails with a 403, see Troubleshooting.

Because the exit code never depends on how many tests were found, the command can't gate a merge on a count. That's deliberate: the result is information for a reviewer, not a pass or a fail.

Sample result

With --output markdown, the command writes a report you can post as is:

## Test impact analysis (advisory)

Impacted: 2 · Coverage gaps: 1

**Impacted (2)**
- [Checkout postal code](https://app.mabl.com/workspaces/{workspace-id}/train/tests/{test-id}/current) - _validates_ · Asserts the inline error text while typing a postal code at checkout.
- [Account settings address edit](https://app.mabl.com/workspaces/{workspace-id}/train/tests/{test-id}/current) - Reaches the same validator when a saved address is edited.

**Coverage gaps (1)** — no credible coverage found for these areas; inconclusive, not proof they are untested
- No test covers the saved-payment expiry field.

**Summary** — the analysis agent's own account
> Searched checkout and account settings; the change reaches both.

Two more lines can appear after the Impacted: … · Coverage gaps: … line:

  • When more_may_exist is true: "More relevant tests likely exist beyond the N returned; the list hit the result ceiling.", followed by a note.
  • When run history couldn't be fully resolved: "Run history could not be fully resolved for some tests; treat missing run details as unknown."

Example: GitHub Actions workflow

This workflow runs the analysis on each pull request, builds the change description from the pull request title and body, and posts the result as a pull request comment. The workflow uses the GitHub CLI (gh), which comes preinstalled on GitHub-hosted runners. If you use self-hosted runners, install gh on them first.

  1. In your GitHub repository, store your mabl API key as an encrypted secret named MABL_API_KEY, and the application ID as a repository variable named MABL_APPLICATION_ID.

  2. Add a workflow file to .github/workflows, such as test-impact.yml, with this content:

    name: Test impact analysis

on:
  pull_request:
    types: [opened, synchronize, reopened, edited]

jobs:
  test-impact:
    runs-on: ubuntu-latest
    if: github.event.pull_request.head.repo.full_name == github.repository && github.actor != 'dependabot[bot]'
    permissions:
      contents: read
      pull-requests: write
    steps:
      - name: Analyze test impact
        continue-on-error: true
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          MABL_API_KEY: ${{ secrets.MABL_API_KEY }}
          PR_URL: ${{ github.event.pull_request.html_url }}
          APPLICATION_ID: ${{ vars.MABL_APPLICATION_ID }}
        run: |
          npm install -g @mablhq/mabl-cli
          mabl auth activate-key "$MABL_API_KEY"

          {
            echo "Title: $(gh pr view "$PR_URL" --json title --jq .title)"
            echo
            gh pr view "$PR_URL" --json body --jq .body
          } > change.md

          mabl tests impact \
            --application-id "$APPLICATION_ID" \
            --change-description-file change.md \
            --output json > impact.json

          jq -r '
            "## Test impact analysis (advisory)\n\nImpacted: \(.tests | length) · Coverage gaps: \(.coverage_gaps | length)\n"
            + (if (.tests | length) > 0 then "\n**Impacted**\n" + ([.tests[] | "- " + .test_name + " — " + .context] | join("\n")) else "" end)
            + (if (.coverage_gaps | length) > 0 then "\n\n**Coverage gaps**\n" + ([.coverage_gaps[] | "- " + .] | join("\n")) else "" end)
          ' impact.json > impact.md

          gh pr comment "$PR_URL" --body-file impact.md

  3. Open a pull request, or push to one that's already open. When the job finishes, the analysis appears as a comment on the pull request.

If you add the workflow as is, it behaves as follows. You can change any of this to fit your pipeline.

  • It runs again when someone edits the pull request. By default, GitHub runs pull request workflows only when a pull request is opened, reopened, or gets new commits. This workflow builds the change description from the pull request title and body, so it also listens for edited. That way, if someone updates the description, the analysis runs again and the comment reflects the latest version.
  • It doesn't fail the job, even if the analysis can't run. The analysis step never fails because of what it finds. It fails only when something goes wrong, such as an API key that doesn't work. Because of continue-on-error: true, even those errors leave the job green. If you'd rather see a red job when the analysis can't run, remove that line.
  • It skips pull requests from forks and Dependabot. Those pull requests can't read your repository secrets, so they can't use your mabl API key. The job's if: condition skips them instead of letting them fail.

Example: Notify your team

Add this as a second step in the same job, after "Analyze test impact." It reads the impact.json that step already saved rather than running the analysis again, so one analysis feeds both the pull request comment and the team notification:

      - name: Notify the team
        continue-on-error: true
        env:
          WEBHOOK_URL: ${{ secrets.TEAM_WEBHOOK_URL }}
        run: |
          summary=$(jq -r '
            "Test impact: \(.tests | length) impacted tests, \(.coverage_gaps | length) coverage gaps"
            + ([.tests[0:3][] | "\n- " + .test_name] | add // "")' impact.json)

          jq -n --arg text "$summary" '{text: $text}' \
            | curl -sS "$WEBHOOK_URL" -H 'Content-Type: application/json' -d @-

Slack incoming webhooks and Microsoft Teams incoming webhooks both accept a JSON body with a text field. Teams workflow webhooks expect a different payload, so check the shape your webhook wants before you wire this up.

Run the impacted tests

To run the tests an analysis returns, pass their IDs to mabl deployments create, up to 50. mabl runs them together as one plan run in a deployment event, and the results appear under Results > By deployment alongside your other deployments.

Run the analysis with --output json, then pass each test's test_invariant_id to --test-ids and the analysis session_id to --impact-session-id:

mabl deployments create \
  --environment-id {environment-id} \
  --application-id {application-id} \
  --test-ids {test-id} {test-id} \
  --impact-session-id {session-id} \
  --await-completion

Unlike the analysis, this command reports a result. With --await-completion, it waits for the tests to finish and exits non-zero if any of them fail. With --retry-failed-tests, a test that passes on its rerun counts as passing. Leave out --await-completion if you want the step to start the tests without waiting for them.

For a complete list of options, see mabl deployments create in the mabl CLI command reference. A few options change how the run connects to the analysis:

  • --impact-session-id links the deployment event to the analysis, so you can see what the analysis triggered, and the GitHub check can report on the selection. Every test in --test-ids must be one the analysis returned. If you add any other test, mabl refuses the whole deployment event.

  • --mabl-branch runs the tests on a mabl branch. Use the same branch you passed to mabl tests impact.

  • --revision connects the results to a GitHub check on the pull request, if you use the GitHub integration.

  • --test-credentials gives individual tests the login credentials their plans use, as space-separated {test id}={credentials id} pairs. Run the analysis with --include-plans, then build the pairs from each test's run_context.plan_credentials_id. Some tests don't have one, so skip those:

    jq -r '.tests[] | select(.run_context.plan_credentials_id) | "\(.test_invariant_id)=\(.run_context.plan_credentials_id)"' impact.json

  • --retry-failed-tests reruns failed tests once, on the same deployment event. It doesn't apply to plans matched by --labels, and you can't combine it with --fast-failure.

  • --selection-name names the plan run in your results. It defaults to "Selected tests".

Without --labels, only the impacted tests run. If you add --labels, the plans that match those labels run on the same deployment event.

Example: Add a run step to the GitHub Actions workflow

In the GitHub Actions workflow above, add this step after "Analyze test impact." It reads the impact.json that step saved, so it runs exactly the tests the analysis returned:

      - name: Run the impacted tests
        env:
          ENVIRONMENT_ID: ${{ vars.MABL_ENVIRONMENT_ID }}
          APPLICATION_ID: ${{ vars.MABL_APPLICATION_ID }}
          REVISION: ${{ github.event.pull_request.head.sha }}
        run: |
          if [ ! -s impact.json ] || [ "$(jq '.tests | length' impact.json)" -eq 0 ]; then
            echo "No impacted tests to run."
            exit 0
          fi

          SESSION_ID=$(jq -r '.session_id // empty' impact.json)

          mabl deployments create \
            --environment-id "$ENVIRONMENT_ID" \
            --application-id "$APPLICATION_ID" \
            --test-ids $(jq -r '.tests[].test_invariant_id' impact.json) \
            ${SESSION_ID:+--impact-session-id "$SESSION_ID"} \
            --revision "$REVISION" \
            --await-completion

An analysis result doesn't always include a session_id. The ${SESSION_ID:+...} line passes --impact-session-id only when it does, because the command refuses an empty or null session ID.

Which version of each test runs

Each test runs at the version on the deployment's mabl branch when mabl accepted the deployment event, or at its master version if it has none there. A retry, or a rerun you start later, picks up the latest version on that branch, so a fix you saved in the meantime is included.

Tests that don't run

mabl skips disabled tests and reports them as not run. If every selected test is disabled, mabl refuses the deployment event. mabl also refuses the whole deployment event, and runs nothing, in cases like these:

  • The selection mixes browser tests and API tests.
  • The selection includes a test type other than browser or API, such as a mobile test.
  • A test has no version on the branch or on master.
  • The environment runs tests from a snapshot of a mabl branch, and a test isn't on that branch. Tests don't fall back to master here.
  • The deployments that match the event resolve to more than one mabl branch.
  • --impact-session-id names an analysis that didn't return one of the tests.
  • A credentials ID in --test-credentials names a test that isn't in --test-ids, or credentials that aren't in your workspace.

Troubleshooting

The analysis fails with a 403

Either your account doesn't have generative AI features, or the API key is missing a scope the analysis requires. Keys from another workspace, or older keys, can be missing that scope. A request made with one of them fails partway through rather than at the start, and a retry won't fix it. Create a new Command Line Interface API key and use that.

Learn more

Related articles