Test case management integrations

Within mabl, it's possible to associate your existing mabl tests with the relevant test case(s) in your test case management (TCM) tool of choice. Although mabl does not integrate directly with these tools, we provide ways to easily retrieve these results to update whichever tool your team is currently using.

Use case

One of the key use cases for integrating with an outside TCM tool is to manage all of your reporting in one, centralized place. This may include manual tests cases, API test cases, and even other automation. For as many user cases there are, there are nearly equally as many tools to choose from. Instead of only supporting one tool, our solution allows you to easily update your test cases within most tools.

Associating outside test cases to mabl tests

The first step in associating your outside test cases is to find the relevant test case within your TCM tool. We'll refer to this Zephyr, TM4J, XRay, etc. test case as the "TCM test case" going forward. You'll want to simply copy the ID of this TCM test case once you've found it.

Once you have the ID copied, head over to mabl and find the relevant mabl test you want to associate. Edit the mabl test from the details page, like you would edit the name or add a label. Find the "Test case ID(s)" section and paste the TCM test case's ID in the field and click save.

If you want to associate multiple TCM test cases to one mabl test, as many manual tests may make up one end-to-end automation test, simply follow the steps above and separate each ID with a , and space as shown in the image above.

Associating TCM test cases when training new mabl tests

We currently only support associating existing mabl tests with TCM test cases, meaning you won't be able to add a TCM test case ID to a mabl test when you train it for the first time. Once you've saved your mabl test, simply follow the steps above.

Retrieving test results

There are three main ways to retrieve your test results from mabl that you can use to integrate with your TCM tool. Two involve processing a JSON detailing specifics about your test runs, the last involves integrating mabl in your CI/CD pipeline via a native plugin, such as Jenkins.

Processing the JSON

Since most TCM solutions won't process the entire JSON response from the API, you'll want to include in your script a process to modify the results and send them directly to your TCM tool in a readable way. Although each TCM tool is different, you'll likely need to retrieve and send a couple of common things:

  • success: true or false
  • test_cases: the test case ID(s) you saved to the test earlier

Via API

Using our deployment events and execution results API, you can trigger mabl deployments and programmatically retrieve their results in JSON format. Note: this method will only update tests results triggered on deployment, not ad hoc results or plans running on a schedule.

Once you've followed the steps above to integrate mabl tests with your TCM test cases, you'll start seeing test_case metadata in the above execution results API response. By using a custom script, such as this sample script, you can easily trigger a deployment then query the results. Here's a sample response:

{
  "plan_execution_metrics": {
    "total": 1,
    "passed": 1,
    "failed": 0
  },
  "journey_execution_metrics": {
    "total": 1,
    "passed": 1,
    "failed": 0
  },
  "executions": [
    {
      "status": "succeeded",
      "success": true,
      "plan": {
        "id": "c953e3e3-a195-4b36-bdf8-49ec73ee3d35",
        "name": "Verify home page load and login",
        "label": "regression",
        "href": "https://api.mabl.com/schedule/runPolicy/c953e3e3-a195-4b36-bdf8-49ec73ee3d35",
        "app_href": "https://app.mabl.com/workspaces/5bf0ebb1-f158-4c02-9e71-1be9f6ce7d17/test/plans/c953e3e3-a195-4b36-bdf8-49ec73ee3d35"
      },
      "plan_execution": {
        "id": "I9wjeolJE-iuvOmCSF28Lw-pe",
        "status": "succeeded",
        "href": "https://api.mabl.com/execution/runPolicyExecution/c953e3e3-a195-4b36-bdf8-49ec73ee3d35"
      },
      "journeys": [
        {
          "id": "6b194f04-1ea0-45f1-8a1a-04c9338067d9:0",
          "name": "Visit home page",
          "href": "https://api.mabl.com/execution/runPolicyExecution/I9wjeolJE-iuvOmCSF28Lw-pe/testScriptExecution/6b194f04-1ea0-45f1-8a1a-04c9338067d9:0",
          "app_href": "https://app.mabl.com/workspaces/5bf0ebb1-f158-4c02-9e71-1be9f6ce7d17/test/plan-executions/I9wjeolJE-iuvOmCSF28Lw-pe/journeys/6b194f04-1ea0-45f1-8a1a-04c9338067d9:0"
        }
      ],
      "journey_executions": [
        {
          "journey_id": "6b194f04-1ea0-45f1-8a1a-04c9338067d9:0",
          "journey_execution_id": "6b194f04-1ea0-45f1-8a1a-04c9338067d9:0",
          "status": "completed",
          "success": true,
          "href": "https://api.mabl.com/test/journey/6b194f04-1ea0-45f1-8a1a-04c9338067d9:0",
          "app_href": "https://app.mabl.com/workspaces/5bf0ebb1-f158-4c02-9e71-1be9f6ce7d17/train/journeys/6b194f04-1ea0-45f1-8a1a-04c9338067d9:0",
          "test_cases": [
            {
              "id": "MABL-1234"
            },
            {
              "id": "MABL-999"
            }
          ]
        }
      ],
      "start_time": 1522777753987,
      "stop_time": 1522777997841
    }
  ]
}

Test cycles

Some test case management tools, such as Zephyr and TM4J, have a concept of "test cycles." While similar to plans in mabl in that they define which tests to run, they aren't exactly the same.

If your TCM tool needs a test cycle to be specified, we recommend modifying your custom script to set the test cycle based on a plan label that you set within mabl, such as: regression, smoke, end-to-end. You can find the plan's label in the JSON above.

This will allow you to make sure your results are being assigned to the correct test cycle every time, even if you change the labels associated to a plan.

Via Webhook

Using post-execution webhooks, you can easily send your test results with relevant test case IDs for every test run. You'll need to have a target URL to send the payload, specifically one that mabl will call with a JSON payload shown below.

Once you've followed the steps above to integrate mabl tests with your TCM test cases, you'll start seeing test_case metadata in the below post-execution webhook payload.

{
  "status": "succeeded",
  "status_cause": null,
  "success": true,
  "plan": {
    "id": "4ba4550d-df2b-4df2-8799-46c44dbb4b9e",
    "name": "Print click and assert Plan",
    "labels": [ "apple", "zebra"],
    "href": "https://api.mabl.com/v1/schedule/runPolicy/4ba4550d-df2b-4df2-8799-46c44dbb4b9e",
    "tags": []
  },
  "plan_execution": {
    "status": "succeeded",
    "status_cause": null,
    "id": "d5b96d02-c45d-4d08-8684-de55411bcede",
    "href": "https://api.mabl.com/v1/execution/runPolicyExecution/d5b96d02-c45d-4d08-8684-de55411bcede"
  },
  "journeys": [
    {
      "id": "c221c1f8-aa52-4fdb-a1bd-5ee5b408ac9f:0",
      "name": "Print, then assert",
      "labels": [ "apple", "zebra"],
      "href": "https://api.mabl.com/v1/test/journey/c221c1f8-aa52-4fdb-a1bd-5ee5b408ac9f:0",
      "tags": [
        "training"
      ]
    }
  ],
  "journey_executions": [
    {
      "status": "completed",
      "status_cause": null,
      "success": true,
      "name": "Print, then assert",
      "journey_id": "c221c1f8-aa52-4fdb-a1bd-5ee5b408ac9f:0",
      "journey_execution_id": "c221c1f8-aa52-4fdb-a1bd-5ee5b408ac9f:0",
      "start_time": 1517612117504,
      "stop_time": 1517612146280,
      "href": "https://api.mabl.com/v1/execution/runPolicyExecution/4ba4550d-df2b-4df2-8799-46c44dbb4b9e/testScriptExecution/c221c1f8-aa52-4fdb-a1bd-5ee5b408ac9f:0",
      "results_href": "https://app.mabl.com/workspaces/3627972b-a762-4a87-a3fb-97cde3e9c414/test/plan-executions/d5b96d02-c45d-4d08-8684-de55411bcede/journeys/c221c1f8-aa52-4fdb-a1bd-5ee5b408ac9f:0",
      "test_cases": [
        {
          "id": "MABL-1234"
        },
        {
          "id": "MABL-999"
        }
      ]
    }
  ],
  "start_time": 1517612110113,
  "stop_time": 1517612146368,
  "tags": [],
  "results": [
    {
      "success": true,
      "name": "Print, then assert",
      "results_href": "https://app.mabl.com/workspaces/3627972b-a762-4a87-a3fb-97cde3e9c414/test/plan-executions/d5b96d02-c45d-4d08-8684-de55411bcede/journeys/c221c1f8-aa52-4fdb-a1bd-5ee5b408ac9f:0"
    }
  ],
  "message_version": "1.0.0"
}

Via CI/CD integrations

The Jenkins and Bamboo integrations enhance the standard JUnit report with information on test cases that are associated with a given test execution.

Test Case IDs in the JUnit Report

There are two sections in the report file that now include test case IDs.

The Properties element section within each TestSuite element may include any of the following property name/value pairs:

Property Name
Property Value

completed-test-cases

The comma-separated list of test case IDs that are associated with passing tests in mabl.

failed-test-cases

The comma-separated list of test case IDs that are associated with failing tests in mabl.

skipped-test-cases

The comma-separated list of test case IDs that are associated with skipped tests in mabl. Tests may be skipped if a dependent test has failed.

In addition, each TestCase element now includes a non-standard Properties element with the requirement key. The value of this key is a comma-separated list of test case IDs. This format is accepted by the XRay application.

Additional CI/CD integrations

Currently, you can only retrieve TCM test case IDs from the results by using the above two methods. We're working on expanding support for integrations such as CircleCI, and more to have access to this data.

Updated 24 days ago


Test case management integrations


Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.