TestRail integration

With the TestRail integration, you can link mabl tests to TestRail cases and record plan run results in TestRail without maintaining a custom script. Your team keeps managing test cases and reviewing results in TestRail, while mabl keeps TestRail current: each completed mabl plan run becomes a TestRail run, with a report of exactly which cases were recorded and which case links were skipped.

This article explains how to set up the TestRail integration, link TestRail cases to mabl tests, and sync plan run results.

• Set up the integration
• Link TestRail cases to mabl tests
• Sync plan run results to TestRail
• When a sync is incomplete or fails

Set up the integration

Gather your TestRail credentials

Before you add the integration in mabl, collect the following details from TestRail:

• The URL of your TestRail Cloud or TestRail Server instance. The instance must be reachable over HTTPS.
• The email address and API key of a TestRail user. To create an API key, open My Settings > API Keys in TestRail.
• The TestRail project whose cases you want to link. If the project uses multiple test suites, you can also note the numeric ID of the suite you want to target.

Add the integration in mabl

After collecting your TestRail credentials, you can configure the integration in mabl:

1. In the mabl app, go to the integrations page: Settings > Integrations.
2. In the Available integrations section, click on the + Setup button for TestRail.
3. Give the integration a name. The default name is "TestRail integration".
4. Under Setup, leave Integration enabled on. Check Automatically sync plan run results if you want mabl to record every completed plan run in TestRail on its own. You can also turn it on later. See Automatic sync for details.
5. Enter the TestRail URL, the Email of the API user, and the API key. If you enter a bare host such as your-instance.testrail.io, mabl adds https:// for you. TestRail URLs must use HTTPS.
6. Click Test connection. mabl checks the credentials against TestRail and, on success, loads the projects that the user can see.
7. Under Configure, choose the Project. If the project is split into several test suites, a Suite ID field appears. Enter the numeric ID of one suite to search it every time, or leave the field empty if you prefer to choose the suite each time you search for a case.
8. Click Save.

The integration appears in the Active integrations section with its status and host URL.

Each TestRail integration connects to one TestRail project. To link cases from another project or another TestRail instance, add a second integration and give it a unique name. When more than one integration is enabled, the case picker on a test lets you choose which instance to search, and a synced plan run gets a separate TestRail run for each integration whose cases it touched.

Edit, disable, or remove the integration

To edit the integration, find it in the Active integrations section and click on the pencil icon. mabl never shows the saved API key again; the field reads "API key (saved)". If you save without entering a key, mabl keeps the stored one. Test connection checks the saved key against the saved URL and email, so to test the connection after changing the URL or email, enter the API key again.

To pause the integration without removing it, turn off Integration enabled and click Save. While the integration is disabled, mabl doesn't link tests to TestRail cases or sync results.

To remove the integration, click on the trash icon in its row and confirm. Case IDs already stored on tests are kept as plain text. They stop linking to TestRail, and results stop syncing. If you add a TestRail integration again later, the stored IDs link to TestRail cases again.

Link TestRail cases to mabl tests

Link mabl tests to TestRail cases in the Test cases section of the test information dialog. Plan runs record results against the linked cases.

The section also accepts IDs from other systems. Choose Other ID in Add from to save an ID exactly as you type it. mabl doesn't send other IDs to TestRail.

Add TestRail cases to a mabl test

1. On the test details page, click on the pencil icon to update test information.
2. In the Test cases section, select your TestRail instance.
3. Find your test case in one of two ways:
   • Start typing part of the case title. mabl searches the project that the integration is configured for and groups the results by suite and section. Select a result.
   • Type the case ID and press Enter. C123, c123, and c0123 all refer to case 123.
4. Click Save.

Each linked case appears as a chip with the TestRail logo and the case title. Click on the title to open the case in TestRail. A test can link up to 20 cases, and near the limit the heading shows how many are in use, such as Linked (18 of 20).

On the test details page, linked TestRail IDs appear beside the test's other details under the TestRail logo. Click on an ID to open the case in TestRail.

Multiple integrations and suites

If more than one TestRail integration is enabled, each instance appears separately in Add from, named by its host. Choose the instance you want to search.

If the TestRail project is split into several test suites and the integration doesn't target one of them, a Suite selector also appears. Choose the suite to search. TestRail searches for cases within a suite, so a title search can't run until you pick one.

You can skip the Suite selector by typing the case ID instead of searching by title. Case IDs are unique across a TestRail instance, so mabl can find C123 without knowing which suite it belongs to.

Remove a case link

On the test details page, click on the pencil icon to update test information, click on the × on the case's chip, and click Save.

Create a test from a TestRail case

If a TestRail case already describes what a test should do, you can hand that case to the mabl agent when building new tests. When the workspace has an enabled TestRail integration, the + menu below the prompt on the Build tests page includes a Case option.

1. Go to Build tests and, on the New test or Edit test tab, click + below the prompt and choose Case.
2. Search TestRail by case title or case ID. Each picked case appears as a chip. You can add up to five cases.
3. Describe anything the case doesn't cover, such as test data or the starting state, and send the prompt.

The agent reads each case's preconditions, steps, and expected results and uses them as the specification for the test outline. If it can't read a case in full, the outline says which case and doesn't claim to cover it.

The search covers the TestRail project that the integration is configured for. If that project has several test suites, the integration must target one suite. Otherwise the Case option can't search it, and the picker tells you to configure a suite on the integration.

Automatic linking

If you build a new test from exactly one TestRail case, mabl links the test to that case when the agent saves it so that future plan runs record results against it. If you reference several cases when building a new mabl test, you'll need to add the links yourself in the Test cases section.

Sync plan run results to TestRail

In TestRail, a run is a record of one pass through a set of cases. A plan run is the equivalent event in mabl, so mabl records results in TestRail at the plan run level. When a mabl plan run syncs, whether manually or automatically, it creates a corresponding run in TestRail and records a result for each case linked to a test in the plan.

A plan doesn't need to be fully linked. mabl tests without a linked TestRail case run as usual in mabl, but they aren't recorded in TestRail and aren't listed as skipped.

The TestRail button in the plan run header shows whether the run's results are in TestRail. Click on the button to check the sync status and to sync the results. If a sync doesn't record everything you expected, see When a sync is incomplete or fails.

Sync a plan run manually

1. Open the plan run output page and click on the TestRail button.
2. Click Sync to TestRail.
3. The button reads "Syncing to TestRail" while the sync runs, then shows the result. The Results went to section of the popover links to the TestRail run that mabl created.

Automatic sync

With automatic sync on, mabl records every completed plan run in TestRail without anyone clicking Sync.

1. Go to Settings > Integrations and click on the pencil icon for the TestRail integration.
2. Under Setup, check Automatically sync plan run results.
3. Click Save.

From the next plan run on, the TestRail button moves from "Waiting for the run to finish" to "Syncing to TestRail" to "TestRail" on its own, usually within a minute of the run completing. The popover shows the TestRail run it created, exactly as it does for a manual sync. Manual Sync to TestRail and Sync again stay available.

To turn automatic sync off, uncheck Automatically sync plan run results in the integration settings and click Save. Runs that already synced stay in TestRail. New runs show "TestRail — not synced" until someone syncs them manually.

What mabl records in TestRail

Each sync creates a TestRail run in the integration's project, named after the mabl plan and plan run. For each TestRail case linked to a test in the plan, mabl adds one result to that run:

• Passed mabl tests are recorded as Passed, and failed tests as Failed.
• TestRail has no built-in Skipped status. If you have added a custom Skipped status to your TestRail instance, mabl uses it for skipped or stopped tests. Otherwise, those tests are reported in the popover as "no result to record".
• Each result's comment starts with "Result reported by mabl" and names the mabl test behind it.
• If several mabl tests link the same case, TestRail receives one result with the worst status among them, and the comment names every test.

Sync again posts the run's results a second time, which is useful if someone edited them in TestRail. Re-syncing doesn't fix a case link that was skipped for a reason on the mabl side. A corrected case ID is picked up by the next plan run.

What isn't synced

• Ad hoc runs of a single test. Only plan runs are recorded in TestRail.
• Cancelled or stopped plan runs.
• Tests that are disabled in the plan, or that have no linked TestRail case. They produce no result in TestRail and aren't listed as skipped.
• IDs for other test case management systems. They're left for the system they belong to.

When a sync is incomplete or fails

Three statuses on the TestRail button mean that a run needs attention:

• TestRail — N links skipped: the run is synced, but some case links couldn't be recorded. Click on the button and review the Skipped section of the popover. Each skipped link is listed with a reason. See Skipped case links.
• TestRail sync failed: the sync stopped before it finished. The popover says how many cases were recorded before it stopped and what TestRail reported. Click Sync again. If it fails the same way, check the integration settings.
• TestRail sync status unavailable: mabl couldn't check whether this run was synced. Reload the page, or sync the run.

The plan's run history and the Plans page show the same state as Sync incomplete or Sync failed, so you can spot a run that needs attention without opening it.

Skipped case links

When a case link couldn't be recorded, the popover lists it with one of the following reasons:

Sync again only helps when the cause was on the TestRail side. A case ID that you correct on the test is picked up by the next plan run.