Skip to main content

Test impact analysis in CI/CD

Molly Exten
  • Updated

Surface coverage gaps and affected tests before shipping code. By adding test impact analysis (TIA) to your CI pipeline, your team gets automated PR comments highlighting these insights, all without risking broken builds or blocking merges.

Follow the steps below to integrate TIA into your CI workflow.

Want to try TIA locally?

If you work with an AI coding agent, the mabl test impact skill runs the same analysis from your editor, describes the change for you, and runs the tests it selects.

Prerequisites

Before you add test impact analysis to a pipeline, gather the following:

  • A mabl API key. We recommend a "Command Line Interface" API key. See managing API keys.
  • The ID of the application you want to analyze. See mabl resource IDs.
  • The mabl CLI installed in your pipeline. Install it globally with npm install -g @mablhq/mabl-cli, as the GitHub Actions example below does, or as a dev dependency with npm install --save-dev @mablhq/mabl-cli. With a dev dependency, run commands as npx @mablhq/mabl-cli instead of mabl.

Add test impact analysis to a GitHub Actions workflow

This workflow runs the analysis on each pull request, builds the change description from the pull request title and body, and posts the result as a pull request comment. The workflow uses the GitHub CLI (gh), which comes preinstalled on GitHub-hosted runners. If you use self-hosted runners, install gh on them first.

  1. In your GitHub repository, store your mabl API key as an encrypted secret named MABL_API_KEY, and the application ID as a repository variable named MABL_APPLICATION_ID.

  2. Add a workflow file to .github/workflows, such as test-impact.yml, with this content:

    name: Test impact analysis

on:
  pull_request:
    types: [opened, synchronize, reopened, edited]

jobs:
  test-impact:
    runs-on: ubuntu-latest
    if: github.event.pull_request.head.repo.full_name == github.repository && github.actor != 'dependabot[bot]'
    permissions:
      contents: read
      pull-requests: write
    steps:
      - name: Analyze test impact
        continue-on-error: true
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          MABL_API_KEY: ${{ secrets.MABL_API_KEY }}
          PR_URL: ${{ github.event.pull_request.html_url }}
          APPLICATION_ID: ${{ vars.MABL_APPLICATION_ID }}
        run: |
          npm install -g @mablhq/mabl-cli
          mabl auth activate-key "$MABL_API_KEY"

          {
            echo "Title: $(gh pr view "$PR_URL" --json title --jq .title)"
            echo
            gh pr view "$PR_URL" --json body --jq .body
          } > change.md

          mabl tests impact \
            --application-id "$APPLICATION_ID" \
            --change-description-file change.md \
            --output json > impact.json

          jq -r '
            "## Test impact analysis (advisory)\n\nImpacted: \(.tests | length) · Coverage gaps: \(.coverage_gaps | length)\n"
            + (if (.tests | length) > 0 then "\n**Impacted**\n" + ([.tests[] | "- " + .test_name + " — " + .context] | join("\n")) else "" end)
            + (if (.coverage_gaps | length) > 0 then "\n\n**Coverage gaps**\n" + ([.coverage_gaps[] | "- " + .] | join("\n")) else "" end)
          ' impact.json > impact.md

          gh pr comment "$PR_URL" --body-file impact.md

  3. Open a pull request, or push to one that's already open. When the job finishes, the analysis appears as a comment on the pull request.

If you add the workflow as is, it behaves as follows. You can change any of this to fit your pipeline.

  • It runs again when someone edits the pull request. By default, GitHub runs pull request workflows only when a pull request is opened, reopened, or gets new commits. This workflow builds the change description from the pull request title and body, so it also listens for edited. That way, if someone updates the description, the analysis runs again and the comment reflects the latest version.
  • It doesn't fail the job, even if the analysis can't run. The analysis step never fails because of what it finds. It fails only when something goes wrong, such as an API key that doesn't work. Because of continue-on-error: true, even those errors leave the job green. If you'd rather see a red job when the analysis can't run, remove that line.
  • It skips pull requests from forks and Dependabot. Those pull requests can't read your repository secrets, so they can't use your mabl API key. The job's if: condition skips them instead of letting them fail.

Notify your team

Add this as a second step in the same job, after "Analyze test impact." It reads the impact.json that step already saved rather than running the analysis again, so one analysis feeds both the pull request comment and the team notification:

      - name: Notify the team
        continue-on-error: true
        env:
          WEBHOOK_URL: ${{ secrets.TEAM_WEBHOOK_URL }}
        run: |
          summary=$(jq -r '
            "Test impact: \(.tests | length) impacted tests, \(.coverage_gaps | length) coverage gaps"
            + ([.tests[0:3][] | "\n- " + .test_name] | add // "")' impact.json)

          jq -n --arg text "$summary" '{text: $text}' \
            | curl -sS "$WEBHOOK_URL" -H 'Content-Type: application/json' -d @-

Slack incoming webhooks and Microsoft Teams incoming webhooks both accept a JSON body with a text field. Teams workflow webhooks expect a different payload, so check the shape your webhook wants before you wire this up.

Run the analysis from other CI tools

In any CI job that can run the mabl CLI, authenticate with your API key, then run mabl tests impact:

mabl auth activate-key "$MABL_API_KEY"

mabl tests impact \
  --application-id {application-id} \
  --change-description-file change.md \
  --output markdown

Writing the description to a file keeps multi-line text out of your shell quoting. For a one-liner, pass it inline with --change-description instead.

Use --output markdown for a report you can post as is, or --output json when a later step reads the result. For every option, see mabl tests impact in the mabl CLI command reference.

Keep the default --timeout. A shorter timeout can give up on an analysis that was about to finish, and a longer one doesn't help, because the analysis itself stops at about five minutes.

Exit codes

The command exits 0 whenever an analysis comes back, including when the analysis finds no impacted tests at all. It exits non-zero only when the call itself fails: an API key it can't authenticate, an application ID it can't resolve, a timeout, or an account without generative AI features. If the command fails with a 403, see Troubleshooting.

Because the exit code never depends on how many tests were found, the command can't gate a merge on a count. That's deliberate: the result is information for a reviewer, not a pass or a fail.

Sample result

With --output markdown, the command writes a report you can post as-is:

## Test impact analysis (advisory)

Impacted: 2 · Coverage gaps: 1

**Impacted (2)**
- [Checkout postal code](https://app.mabl.com/workspaces/{workspace-id}/train/tests/{test-id}/current) - _validates_ · Asserts the inline error text while typing a postal code at checkout.
- [Account settings address edit](https://app.mabl.com/workspaces/{workspace-id}/train/tests/{test-id}/current) - Reaches the same validator when a saved address is edited.

**Coverage gaps (1)** — no credible coverage found for these areas; inconclusive, not proof they are untested
- No test covers the saved-payment expiry field.

**Summary** — the analysis agent's own account
> Searched checkout and account settings; the change reaches both.

Two more lines can appear after the Impacted: … · Coverage gaps: … line:

  • When more_may_exist is true: "More relevant tests likely exist beyond the N returned; the list hit the result ceiling.", followed by a note.
  • When run history couldn't be fully resolved: "Run history could not be fully resolved for some tests; treat missing run details as unknown."

Writing change descriptions

Unlike the mabl test impact skill, which reads your code changes, TIA in CI can't see the diff. It only knows what the change description tells it, which in the GitHub Actions workflow is the pull request title and body. TIA matches that text against the content of your tests, such as step descriptions, assertions, and page names. The closer your description is to the way your tests describe your product, the more of the right tests it finds.

To get the most out of TIA in CI, adhere to the following guidelines:

  • Start with the pull request. A title and description written for a human reviewer are already most of a good change description. The GitHub Actions example above uses the PR title and body directly.
  • Describe the change in product vocabulary, not implementation detail. A description like "Memoized the validation hook in FieldGroup.tsx" won't retrieve anything because mabl tests don't describe files or classes. A better description would be "Validation now runs on each keystroke instead of on blur, so error text appears while the user is still typing."
  • For shared code, name every user-facing area the change reaches. For example, if a validator shows up at checkout and wherever a customer edits a saved address, call that out in the description.

A small map from code paths to product areas, kept in your pipeline, recovers the product vocabulary that a diff alone doesn't have:

apps/web/src/checkout/**  = checkout and payment
apps/web/src/account/**   = account settings and saved payment methods
apps/web/src/search/**    = product search and filtering
packages/forms/**         = form validation

A change that touches packages/forms/** and apps/web/src/checkout/** then adds a line to the description such as: "Product areas touched: form validation, checkout and payment."

Troubleshooting

The analysis fails with a 403

Either your account doesn't have generative AI features, or the API key is missing a scope the analysis requires. Keys from another workspace, or older keys, can be missing that scope. A request made with one of them fails partway through rather than at the start, and a retry won't fix it. Create a new Command Line Interface API key and use that.

Learn more