With the Azure Boards integration, mabl can record plan run results in Azure Test Plans, next to your team's manual results. Your test plan's progress charts and reports then include mabl's automated testing, without a pipeline step or custom scripts. mabl can also keep the Automation status of linked Test Cases up to date.
This article explains how to link mabl tests to Test Cases, record plan run results in a test plan, and choose when mabl marks Test Cases as automated.
Before you start
- A connected Azure Boards integration. The account it connects with needs Basic access and the Contributor role, the same as for filing work items.
- A test plan in the integration's project, with the Test Cases your mabl tests link to added to it. Creating test plans and adding Test Cases to them in Azure DevOps requires the Basic + Test Plans access level, so someone on your team with that access may need to set them up.
Link mabl tests to Test Cases
Link mabl tests to Test Cases in the Test cases section of the test information dialog, the same way you link cases for other test case management integrations.
- On the test details page, click on the pencil icon to update test information.
- In the Test cases section, under Add test cases, choose the Azure DevOps project in Add from.
- Search Test Cases by title, or enter the Test Case's work item ID, such as
123orAB#123, or its URL. - Click Save.
With your Azure DevOps project selected in Add from, 123 and AB#123 both link work item 123. Elsewhere, such as under Other ID, a bare number is read as a TestRail case ID, so use the AB# prefix.
If your workspace has Azure Boards integrations in more than one Azure DevOps organization, link Test Cases by URL, because an AB# ID can't sync. Picking a Test Case from search saves the right form. If two integrations connect the same Azure DevOps project, links to that project show cannot sync until you disable one of the integrations.
Only work items of the Test Case type receive results. If you link another type, such as a user story, mabl marks it not a Test Case in the section and plan runs skip it.
If you build a test from a single Test Case, mabl links it for you.
Record plan run results
When a plan run finishes, mabl adds a test run to your test plan, named after the mabl plan. It then records a result on each Test Case that the plan's tests link to. mabl tests without a linked Test Case run as usual, but they aren't recorded in Azure Test Plans.
Turn on results sync
- Go to Settings > Integrations and click on the pencil icon for the Azure Boards integration.
- Under Results sync, check Sync plan run results to Azure Test Plans.
- To record every completed plan run without syncing it by hand, check Sync automatically when a plan run finishes. Ad hoc runs aren't synced automatically.
- Choose the Default test plan that mabl records results in, such as "Release 3.2". Keep in mind that you can also configure mabl plans to choose their own test plan.
- Optional: enter Configuration IDs. See Choose configurations.
- Click Save.
Choose configurations
In Azure Test Plans, a configuration describes an environment you test on, such as "Chrome on Windows" or "Safari on macOS". A Test Case gets one result slot for each configuration assigned to it in the test plan. By default, mabl records its result in every slot.
If your team uses configurations to separate browsers or devices, enter the IDs of the configurations that mabl's runs represent, separated by commas, such as 12, 14. mabl then records results only for those configurations. If your test plan uses only the default configuration, leave the field empty.
Choose a test plan for a mabl plan
The integration settings record all mabl results in the default test plan. If you want to record results in different test plans, you can update this in mabl plan settings:
- Click the pencil icon to edit the mabl plan.
- Under The basics, find the Azure test plan field for your integration, such as Azure test plan (Azure Boards). The field appears for each Azure Boards integration with results sync turned on.
- Choose a test plan from the integration's project, such as "Sprint 42". To go back to the integration's default test plan, choose Integration default.
- Click Save.
Keep in mind the following when you change a mabl plan's test plan or the integration's default test plan:
- Plan runs that finish after you save record their results in the new test plan. Results already recorded stay where they are.
- Syncing a plan run again records it in the test plan its mabl plan chose when it ran. A run that used the integration default goes to the current default test plan, and Test Cases that aren't in it are skipped as not in the test plan.
Sync a plan run manually
If automatic sync is off, or you want to record a plan run that finished before you turned it on, you can sync it from the plan run page. Syncing needs the editor role or higher in the workspace, or an API key with write access.
- Open the plan run output page and click on the Azure Test Plans button in the header.
- Click Sync to Azure Test Plans.
- The button reads "Syncing to Azure Test Plans" while the sync runs, then shows the result. The Results went to section of the popover links to the test run in Azure Test Plans.
A sync fails in the following cases:
- The plan run is cancelled or stopped.
- The plan run finished more than 180 days ago.
- The plan run links more than 1,000 Test Cases, or more than 10,000 result slots across their configurations. To stay under the limit, record fewer configurations or split the plan.
What mabl records
For each Test Case linked to a test in the plan, mabl records one result in the plan run's test run:
- Outcome: Passed, Failed, or Not Executed. Not Executed means every linked test was skipped or stopped. If several mabl tests link the same Test Case, one failing test marks the result as failed, and the comment says how many runs it covers.
- Link to mabl: each result links to the mabl test run.
- Failure details: a failed result also includes mabl's failure analysis, when your workspace has one, and a screenshot of the application's final state, which is also attached to the result. Azure DevOps limits the result comment to 1,000 characters, so a long failure analysis is shortened. The full analysis is on the mabl test run.
Failure analysis and screenshots are visible to everyone with access to the Azure DevOps project, which may be a wider group than your mabl workspace.
Sync again records only the Test Cases that this plan run has no result for yet, so syncing twice doesn't create duplicate results.
When a sync is incomplete or fails
Three statuses on the Azure Test Plans button mean that a run needs attention:
- Azure Test Plans — N links skipped: the run is synced, but some case links couldn't be recorded. Click on the button and review the Skipped section of the popover. See Skipped case links.
- Azure Test Plans sync failed: the sync stopped before it finished. The popover says how many cases were recorded before it stopped. Click Sync again. If it fails the same way, check the integration settings. A sync also fails when results sync is turned off or the test plan no longer exists. The message then names the test plan and whether the integration or the mabl plan chose it. If you move the integration to another Azure DevOps project, mabl plans that chose a test plan in the old project fail to sync, and the message names both projects. Choose a test plan from the new project on each of those mabl plans.
- Azure Test Plans sync status unavailable: mabl couldn't check whether this run was synced. Reload the page, or sync the run.
The plan's run history shows the same state as Sync incomplete or Sync failed, so you can spot a run that needs attention without opening it.
Skipped case links
When a case link couldn't be recorded, the popover lists it with one of the following reasons:
| Reason | What to do |
|---|---|
| not in the test plan | Add the Test Case to the test plan that the run recorded in. Then sync again. |
| no test point | The Test Case is in the test plan but has no result slot for the configurations mabl records. Assign it a configuration in its suite, or add that configuration's ID to Configuration IDs, or clear the field. Then sync again. |
| not a Test Case | The linked work item is another type, such as a user story. Link the Test Case that covers it instead. |
| not an Azure work item | Fix the link on the test, such as AB#123. The next plan run syncs it. |
| not found in Azure DevOps | Check that the work item exists, that it's in the integration's project, and that the integration's account can see it. In a workspace connected to more than one Azure DevOps organization, link it by URL. Then sync again. |
| another system's link | The ID isn't an Azure work item, so mabl leaves it for the system it belongs to. No action needed. |
Sync again only helps when the cause was on the Azure DevOps side, such as a Test Case added to the test plan. A link that you fix on the test is picked up by the next plan run.
Mark linked Test Cases as automated
Test Cases in Azure DevOps have an Automation status: Not Automated, Planned, or Automated. Teams use it to track how much of a test plan is automated. Azure DevOps normally sets a Test Case to Automated only when a build pipeline associates it with an automated test. mabl can set it for you, based on the mabl tests that link the Test Case.
Automation status is separate from results sync. It doesn't depend on the test plan, and mabl records results on linked Test Cases whichever option you choose.
To choose when mabl updates Automation status:
- Go to Settings > Integrations and click on the pencil icon for the Azure Boards integration.
- Under Test Cases, choose an option:
- Don't change Automation status: mabl leaves Automation status alone.
- When linked: mabl marks a Test Case Automated as soon as a mabl test links it.
- After a passing run: mabl sets a Test Case to Planned when a mabl test links it, then marks it Automated after that test passes a cloud run on the main branch. Only runs from the last 30 days count.
- Click Save.
Changes show up in Azure DevOps within about an hour. A Test Case that mabl marks shows the linking mabl test as its associated automation.
How mabl updates Automation status
- mabl never changes automation set by a pipeline or another tool. If a Test Case is already associated with another automated test, mabl leaves it as it is.
- Automated stays Automated. If a test fails after mabl marks its Test Case Automated, the Test Case stays Automated.
- mabl keeps its marking in place. While a mabl test links a Test Case that mabl marked, mabl keeps it Automated, even if someone changes it in Azure DevOps. Unlink the test to release it.
- Planned only replaces an empty status. With After a passing run, mabl sets Planned only on a Test Case whose status is empty or Not Automated.
- Unlinking resets the Test Case. When no mabl test links a Test Case that mabl marked, mabl sets it back to Not Automated.
- mabl doesn't clear Planned. If you unlink a test before it passes, its Test Case stays Planned until someone changes it in Azure DevOps.
For example, with After a passing run, you link a new mabl test to Test Case AB#123. Within the hour, the Test Case shows Planned. The test passes its first cloud run on the main branch the next day, and the Test Case changes to Automated. A week later, the test fails, and the Test Case stays Automated while mabl records the failed result in the test plan.
Turn off Automation status updates
If you choose Don't change Automation status, mabl stops updating Automation status. Test Cases that mabl already marked stay Automated. When you choose When linked or After a passing run again, mabl resumes updates and removes its marking from Test Cases that no mabl test links.
For example, a team with Don't change Automation status links a mabl test to Test Case AB#456. The Test Case stays Not Automated, but mabl still records a result on it after every plan run.