API-based CI/CD integration – mabl help

Most mabl customers use mabl's existing integrations or the mabl CLI to integrate end-to-end testing into their DevOps pipeline. However, If none of those options fit your needs, you can use the mabl APIs to run end-to-end tests on code deployment.

Recommended reading

Before integrating mabl into your CI/CD pipeline, please review the following resources:

This guide provides you with an example of how to integrate mabl as a step in your CI/CD pipeline using the Deployment Events API:

The first API call triggers all plans associated with a particular environment, application, and/or plan labels in response to a new software deployment event. If your pipeline runs test impact analysis, the same call can run the tests the analysis selects.
The second API call checks the status of the plans that were launched as a result of the first API call.

The provided examples assume that you are using a CI build tool like Jenkins, Gitlab, or CircleCI and a Linux-based container which allows you to write custom build steps in bash with standard tools such as curl and jq. If you are using a different type of build tool you may need to adapt these examples.

Hypothetical CI/CD workflow

Imagine that your team has an automated CI/CD workflow into which you want to integrate end-to-end testing with mabl. For example, consider the following pipeline, which represents part of a CI/CD workflow:

Create deployment event

The first step in adding mabl to your CI/CD flow is to notify mabl when you deploy your application using a mabl deployment event. When you invoke the create deployment event endpoint, mabl searches for all plans associated with the specified environment, application and/or labels that contain a deployment trigger, plus any plans you name by ID, and then executes those plans immediately.

Before you can get started with the Deployment Events API you'll need a few pieces of information:

A deployment trigger API key. (See Managing API keys for more info.)
At least one of the following mabl resource IDs:
- Environment ID
- Application ID
Plan labels or plan IDs: if you want to target specific plans

Building a request

The mabl deployment builder on the APIs page generates the command for you, as either a mabl CLI command or a cURL request. To build the cURL request:

Navigate to Settings > APIs and scroll down to Run tests on deployment.
Select cURL as the command type.
Under Authenticate with, choose the API key for the request. The dropdown lists only keys that can trigger deployments.
Choose the target and add any overrides or deployment details to generate the curl command.

Request fields

Every request must include environment_id, application_id, or both. All other fields are optional. Here is a fuller example of a request body:

{
  "environment_id": "FQWY5iMcfkmqV64YMyrt5w-e",
  "plan_labels": ["smoke"],
  "revision": "937e77b3822a4ae2a38135e08a5d4a8eb6eddfdd",
  "plan_overrides": {
    "web_url": "https://preview-1013.example.com",
    "browser_types": ["chrome", "firefox"]
  }
}

These are the fields most pipelines use:

plan_labels: Run only the plans that have at least one of these labels, such as smoke, instead of every plan for the environment or application.
branch: Run tests from a mabl branch, such as the branch that matches the code you deployed.
revision: The source code revision of the deployment, such as a git commit hash. Send it so you can trace results back to the commit you deployed.
plan_overrides: Change how the triggered plans run for this deployment only. See Plan overrides.
test_selection: Run the tests a test impact analysis selects. See Run the tests test impact analysis selects.

To run specific plans, add their IDs to plan_ids at the top level of the request body, up to 50, such as "plan_ids": ["cLhFR8Fz0Kpsh70zP6EQ3Q-p"]. Without plan labels, only these plans run. With plan labels, they run alongside the plans that match the labels. If any named plan can't run on the deployment event, mabl rejects the request and no plans run. See Run specific plans by ID.

For every field the endpoint accepts, see Create a deployment event in the mabl API reference.

Plan overrides

Use plan_overrides to run the triggered plans with different settings from the ones saved on the plans, without editing the plans. For example, you can point the plans at a preview environment URL, run them on other browsers, or log in with different credentials. The example request above runs the plans against a preview URL in Chrome and Firefox.

Deployment actions

Use actions to have the environment run tests from a snapshot of a mabl branch, or to reset the visual baselines for the tests that run. The API reference lists the fields for each action.

Run the tests test impact analysis selects

If you run test impact analysis in your CI pipeline, a deployment event can run the impacted tests the analysis returns. In test_selection, set:

test_ids: Each test's test_invariant_id, up to 50.
impact_session_id: The analysis session_id.
source: agent, since the analysis chose the tests.

mabl runs the tests together as one plan run on the deployments that match the event's environment, application, or both. For example:

{
  "environment_id": "FQWY5iMcfkmqV64YMyrt5w-e",
  "revision": "937e77b3822a4ae2a38135e08a5d4a8eb6eddfdd",
  "test_selection": {
    "test_ids": ["Xk3pQ7rLs0e2YvNw9Ha1Bg-j"],
    "impact_session_id": "{session-id}",
    "source": "agent",
    "retry_failed_tests": true
  }
}

Without plan_labels, only the selected tests run. If you also send plan_labels, the matching plans run on the same deployment event.

To learn more, see Run the impacted tests. The API reference lists the rest of the test_selection fields.

Triggering a deployment event

When you trigger a deployment event, you get a JSON message back with the deployment event ID and the plans it started. For example:

{
  "id": "OCn2EccmXk9eFvRInTfwkQ-v",
  "environment_id": "FQWY5iMcfkmqV64YMyrt5w-e",
  "workspace_id": "RS2yqNMP9nK9qobpJiZ74Q-w",
  "received_time": 1522777753410,
  "plan_labels": ["smoke"],
  "triggered_plan_run_summaries": [
    {
      "plan_id": "cLhFR8Fz0Kpsh70zP6EQ3Q-p",
      "plan_run_id": "I9wjeolJE-iuvOmCSF28Lw-pr"
    }
  ]
}

The response repeats the request fields you sent, plus fields that mabl sets. These are the ones to look at:

id: The deployment event ID. Use it to get the results of the tests the deployment event triggered.
triggered_plan_run_summaries: The plan_id and plan_run_id of each plan the deployment event started. Check it to confirm the request matched the plans you expected.
test_selection.skipped_tests: If you sent a test_selection, the selected tests that didn't run and why. Today the only reason is disabled. The field is left out of the response when nothing was skipped. It comes back for any test_selection, not only one linked to an analysis.

For tests run with test_selection, triggered_plan_run_summaries includes an entry whose plan_id is the deployment event ID followed by -selection. No saved plan has that ID. It's the plan run mabl creates for the selected tests.

For every response field, see the API reference.

Get results of the deployment event

After creating a deployment event, you can check the status of plans that are launched by using the get a deployment result summary endpoint.

Building a request

The same builder generates the request for the results:

Navigate to Settings > APIs and scroll down to Run tests on deployment.
Select cURL as the command type.
Under Check the results, enter the deployment event ID to generate the curl command.

Getting a response

The get a deployment result summary endpoint returns a JSON object containing details about the status of all plans that were triggered by the deployment event.

Here is a sample response:

{
  "event_status": {
    "succeeded": true,
    "succeeded_first_attempt": true
  },
  "plan_execution_metrics": {
    "total": 1,
    "passed": 1,
    "failed": 0
  },
  "journey_execution_metrics": {
    "total": 1,
    "passed": 1,
    "failed": 0
  },
  "executions": [
    {
      "status": "succeeded",
      "success": true,
      "plan": {
        "id": "c953e3e3-a195-4b36-bdf8-49ec73ee3d35",
        "name": "Verify home page load and login",
        "label": "regression",
        "href": "https://api.mabl.com/schedule/runPolicy/c953e3e3-a195-4b36-bdf8-49ec73ee3d35",
        "app_href": "https://app.mabl.com/workspaces/5bf0ebb1-f158-4c02-9e71-1be9f6ce7d17/test/plans/c953e3e3-a195-4b36-bdf8-49ec73ee3d35"
      },
      "plan_execution": {
        "id": "I9wjeolJE-iuvOmCSF28Lw-pe",
        "status": "succeeded",
        "href": "https://api.mabl.com/execution/runPolicyExecution/c953e3e3-a195-4b36-bdf8-49ec73ee3d35"
      },
      "journeys": [
        {
          "id": "6b194f04-1ea0-45f1-8a1a-04c9338067d9:0",
          "name": "Visit home page",
          "href": "https://api.mabl.com/execution/runPolicyExecution/I9wjeolJE-iuvOmCSF28Lw-pe/testScriptExecution/6b194f04-1ea0-45f1-8a1a-04c9338067d9:0",
          "app_href": "https://app.mabl.com/workspaces/5bf0ebb1-f158-4c02-9e71-1be9f6ce7d17/test/plan-executions/I9wjeolJE-iuvOmCSF28Lw-pe/journeys/6b194f04-1ea0-45f1-8a1a-04c9338067d9:0"
        }
      ],
      "journey_executions": [
        {
          "journey_id": "6b194f04-1ea0-45f1-8a1a-04c9338067d9:0",
          "journey_execution_id": "6b194f04-1ea0-45f1-8a1a-04c9338067d9:0",
          "status": "completed",
          "success": true,
          "href": "https://api.mabl.com/test/journey/6b194f04-1ea0-45f1-8a1a-04c9338067d9:0",
          "app_href": "https://app.mabl.com/workspaces/5bf0ebb1-f158-4c02-9e71-1be9f6ce7d17/train/journeys/6b194f04-1ea0-45f1-8a1a-04c9338067d9:0",
          "test_cases": [
            {
              "id": "MABL-1234"
            },
            {
              "id": "MABL-999"
            }
          ]
        }
      ],
      "start_time": 1522777753987,
      "stop_time": 1522777997841
    }
  ]
}

For most purposes it should be sufficient to process only the summary information under plan_execution_metrics_and journey_execution_metrics. Any output field named app_href contains a link to the mabl UI that may be used to directly view an entity or the results of an execution.

Tests used to be called journeys

The API response still reflects the old term for tests: "journey".

Inserting mabl into your pipeline

One option for inserting mabl into your CI/CD pipeline would be to:

Trigger a deployment event to run certain plans when a deployment occurs to a particular environment.
Poll the endpoint for retrieving the results of a deployment event until all plans have either passed or failed.

Using the hypothetical CI/CD workflow from earlier, it might look something like this:

A flowchart that includes mabl tests as part of the CI/CD workflow

Tying it all together

The following bash script can trigger a deployment event and then poll the executions until they either complete or fail. This sample script takes the API key and environment ID as arguments: sh mabl-deployment-integration.sh api-key env-id.

#!/bin/bash

MABL_API_BASE="https://api.mabl.com"
POLL_SEC=10

if [ "$#" -ne 2 ]; then
    echo "Usage: $0  "
    exit 0
fi

API_KEY="$1"
ENV_ID="$2"

# Create a deployment event, and retrieve its ID:

deployment_event=$(curl -s "${MABL_API_BASE}/events/deployment" -u "key:${API_KEY}" -H 'Content-Type:application/json' -d "{\"environment_id\":\"${ENV_ID}\"}")
event_id=$(echo "${deployment_event}" | jq -r '.id')
echo "Deployment Event ID: ${event_id}"

# Poll the Execution Result API until all Plans have completed

while true; do
    echo "Waiting for executions to complete..."
    sleep ${POLL_SEC}
    results=$(curl -s "${MABL_API_BASE}/execution/result/event/${event_id}" -u "key:${API_KEY}")
    event_status=$(echo "${results}" | jq '.event_status')
    if [ -z "${event_status}" ] || [ "${event_status}" = "{}" ]; then
       continue
    fi

    plan_metrics=$(echo "${results}" | jq '.plan_execution_metrics')
    if [ "${plan_metrics}" != "null" ]; then
        total_plans=$(echo "${plan_metrics}" | jq -r '.total')
        passed_plans=$(echo "${plan_metrics}" | jq -r '.passed')
        failed_plans=$(echo "${plan_metrics}" | jq -r '.failed')
    fi

    succeded=$(echo "${event_status}" | jq -r '.succeeded')
    succeded_with_retries=$(echo "${event_status}" | jq -r '.succeeded_with_retries')
    break
done

# Print summary:

echo
echo "Full Results:"
echo "${results}" | jq
echo
echo "Total plans executed: ${total_plans} succeded: ${passed_plans} failed: ${failed_plans}"
if [ "${succeded}" = "true" ]; then
    if [ "${succeded_with_retries}" = "true" ]; then
       echo "Successful run with retries"
    else
       echo "Successful run"
    fi
    exit 0
else
    echo "One or more plans have failed"
    exit 1
fi

Hypothetical CI/CD workflow

Create deployment event

Building a request

Request fields

Plan overrides

Deployment actions

Run the tests test impact analysis selects

Triggering a deployment event

Get results of the deployment event

Building a request

Getting a response

Inserting mabl into your pipeline

Tying it all together

Related articles