Integrating with test case management tools

Adding mabl test results to an outside test case management (TCM) tool, such as XRay, Zephyr, Practitest, or Testrail, allows you to see the aggregate results of your testing in one centralized place. The results in the TCM tool become a single source of truth and can help teams meet other requirements more easily, such as providing documentation for auditing purposes.

This guide outlines how to set up a custom integration with a TCM tool.

🚧

No native integration

Mabl does not support a native integration with any test case management tools. To set up an integration using the steps below, you or someone on your team should be comfortable with creating and maintaining custom scripts.

Initial setup

Before creating a custom TCM integration, you need to associate mabl tests with their relevant test cases:

1. In your TCM tool, identify the relevant test case and copy its ID.
2. In mabl, find the mabl test that you want to associate with the test case ID. On the Test Details page, click on the pencil icon to edit test metadata.

3. Paste the test case ID in the "Test case ID(s)" and click Save.

To associate the mabl test with multiple test case IDs, separate each ID with a comma and a space.

📘

Adding plan labels

If your TCM tool requires a specific project ID or test cycle to add results, we recommend adding plan labels to the relevant mabl plans.

Plan labels are included in the test result. You can modify your custom integration to set the project ID or test cycle based on the plan label that you set within mabl.

Custom integration

After associating mabl tests with test case IDs, you need to set up a workflow that integrates mabl test results into your TCM tool. At a high level, the workflow for a custom integration consists of three steps:

1. Retrieve test results
2. Parse the data
3. Add results to your TCM tool

The process for integrating test results into your TCM tool depends on how you retrieve results. You can retrieve results in two formats:

• JSON payload
• JUnit report (Jenkins and Bamboo integrations only)

Read on to learn how to integrate these two types of test result formats.

JSON payload

To retrieve mabl test results in a JSON payload, use one of the following methods:

• Post-execution webhook: Configure a webhook URL to receive mabl test results with the relevant test case IDs after test runs complete.
• Deployment event API: Create a deployment event and then poll the deployment result summary endpoint for the results. See a sample script here.

📘

Note

Post-execution webhooks receive results for all types of mabl test runs, including deployment events, plan runs, and ad hoc runs, whereas the deployment result summary endpoint only retrieves results for tests triggered by a deployment event

The JSON payload includes information on status, outcome, test case IDs, and other details for specific plans and tests that ran. Tests are referred to as "journeys" in the JSON payload.

Test case IDs appear as an array of objects within journey_executions.test_cases.

{
  "status": "succeeded",
  "status_cause": null,
  "success": true,
  "plan": {
    "id": "4ba4550d-df2b-4df2-8799-46c44dbb4b9e",
    "name": "Print click and assert Plan",
    "labels": [ "apple", "zebra"],
    "href": "https://api.mabl.com/v1/schedule/runPolicy/4ba4550d-df2b-4df2-8799-46c44dbb4b9e",
    "tags": []
  },
  "plan_execution": {
    "status": "succeeded",
    "status_cause": null,
    "id": "d5b96d02-c45d-4d08-8684-de55411bcede",
    "href": "https://api.mabl.com/v1/execution/runPolicyExecution/d5b96d02-c45d-4d08-8684-de55411bcede"
  },
  "journeys": [
    {
      "id": "c221c1f8-aa52-4fdb-a1bd-5ee5b408ac9f:0",
      "name": "Print, then assert",
      "labels": [ "apple", "zebra"],
      "href": "https://api.mabl.com/v1/test/journey/c221c1f8-aa52-4fdb-a1bd-5ee5b408ac9f:0",
      "tags": [
        "training"
      ]
    }
  ],
  "journey_executions": [
    {
      "status": "completed",
      "status_cause": null,
      "success": true,
      "name": "Print, then assert",
      "journey_id": "c221c1f8-aa52-4fdb-a1bd-5ee5b408ac9f:0",
      "journey_execution_id": "c221c1f8-aa52-4fdb-a1bd-5ee5b408ac9f:0",
      "start_time": 1517612117504,
      "stop_time": 1517612146280,
      "href": "https://api.mabl.com/v1/execution/runPolicyExecution/4ba4550d-df2b-4df2-8799-46c44dbb4b9e/testScriptExecution/c221c1f8-aa52-4fdb-a1bd-5ee5b408ac9f:0",
      "results_href": "https://app.mabl.com/workspaces/3627972b-a762-4a87-a3fb-97cde3e9c414/test/plan-executions/d5b96d02-c45d-4d08-8684-de55411bcede/journeys/c221c1f8-aa52-4fdb-a1bd-5ee5b408ac9f:0",
      "test_cases": [
        {
          "id": "MABL-1234"
        },
        {
          "id": "MABL-999"
        }
      ]
    }
  ],
  "start_time": 1517612110113,
  "stop_time": 1517612146368,
  "tags": [],
  "results": [
    {
      "success": true,
      "name": "Print, then assert",
      "results_href": "https://app.mabl.com/workspaces/3627972b-a762-4a87-a3fb-97cde3e9c414/test/plan-executions/d5b96d02-c45d-4d08-8684-de55411bcede/journeys/c221c1f8-aa52-4fdb-a1bd-5ee5b408ac9f:0"
    }
  ],
  "message_version": "1.0.0"
}

Since most TCM tools cannot process the entire JSON payload, you need to write a parsing script to extract the relevant details. While each TCM tool is different, you likely need to extract a couple of common pieces of metadata before adding results:

• Project ID
• Test case ID(s)
• Status - the test result
• Comment

Refer to your TCM tool's API documentation for more details on which endpoints to use and what information to send.

👍

Reporting API

You can use the batch results endpoint to get data on test results over time. This endpoint is useful if you want to add data on mabl tests that ran before you set up the integration with your TCM tool.

JUnit reports

If your team triggers mabl tests using the Jenkins or Bamboo integrations, you can get test results and associated test case IDs from the JUnit xml report.

The Properties element section within each TestSuite element may include any of the following property name/value pairs:

Each TestCase element also includes a non-standard Properties element with the requirement key. The value of this key is a comma-separated list of test case IDs.

The JUnit xml report format is accepted by most test case management tools, but you may need to configure how this report is ingested to properly map your mabl test results to your test case IDs. Refer to your TCM tool's documentation on importing JUnit results for more details.