.svg "Leapwork")
Automatic creation of bugs in Azure DevOps Server
An important part of running automated test cases is to monitor, inspect, and react to the results. This is best accomplished in the test management and bug tracking system that you use for your daily work.
In this article, you can review three different scenarios on how to trigger and create/update bugs in the Azure DevOps Server, formally known as Visual Studio Team Services (VSTS) or Team Foundation Server (TFS), based on the results of test cases running in LEAPWORK.
Use a plugin for Azure DevOps Servers
Use the LEAPWORK plugin to easily integrate with the Azure DevOps Servers. This extension can be installed from the Visual Studio Marketplace as follows:
- Navigate to the Visual Studio Marketplace.
- Type LEAPWORK in the search bar and select it.
- Follow the instructions on the page.
This LEAPWORK plugin is a community-developed app and can be found on GitHub.
Creat bugs from schedules
To run a collection of test cases in one or more environments (machines, devices, etc.) on a scheduled basis or by triggering test cases through the LEAPWORK REST API, you must define a schedule. In a schedule, you can select one or more test cases from a LEAPWORK project and pair them with one or more environments. Furthermore, you can set the duration and frequency of scheduled test case runs.
In a schedule, you can select Add action and define one or more actions that should be performed during a test case. One example of an action is to send an e-mail when a schedule has ended. Another is to call a specified web service if more than X number of test cases in the schedule has failed.
In this case, a PowerShell action is triggered when the schedule is finished. This action is used to call a local PowerShell script and pass the result of the schedule to the PowerShell script. An example of how to specify this action is shown in the following figure:
The command tells LEAPWORK to run a PowerShell script, c:\scripts\createTFSbugs.ps1.
The following parameters have been added:
- -result: This parameter is read by the PowerShell script and is the result of the entire schedule. To add the result data, right-click in the command field and select Insert token -> RESULTS-JSON.
Warning: The inserted token must be in between
@”
[
and
]
“@
to include the line breaks. This structure ensures that PowerShell transfers the JSON result format correctly.
- -rootPath: This parameter is read by the PowerShell script and is the file path to the folder containing the videos and screenshots recorded during the execution of the test cases.
The PowerShell script, CreateTFSbugs.ps1, will connect to the Azure DevOps Server REST API and create or update bugs based on the failed test cases in the schedule.
To use the PowerShell script, you need to specify a few settings about you Azure DevOps Server installation:
domain: If you are using Visual Studio Online (VSO), insert your visualstudio.com domain name, for example, https://LEAPWORK.visualstudio.com
tfsPath: The path to your project.
pat: Personal Access Token (PAT). You can find more information about setting up a PAT here.
logfile: The path to the integration log file. All log entries are appended to this file.
The script has been tested against VSO, but it’s the same API used for the on-premise installations of Azure DevOps Server. There may be differences in various REST API versions. Refer to Azure DevOps documentation for further information.
Trigger schedule from API and create bugs in Azure DevOps Server
Integrating LEAPWORK with any third-party system through the REST API can be done in a few simple steps. This enables you to easily set up the automatic creation or updating of bugs in Azure DevOps Server when test automation cases fail in LEAPWORK.
Download sample PowerShell script
# LEAPWORK REST API example: Run a schedule, iterate through the results and create bugs in TFS.#
#
# Author: Claus Topholt.
# Function that finds a schedule in LEAPWORK based on a title, runs it and polls for the results.
function RunScheduleAndGetResults($schedule)
{
Write-Host "Getting id for schedule '$schedule'."
# Get the id of the schedule.
$runScheduleId = "";
$headers = @{}
$headers.Add("AccessKey","bTyGAd0UGL70JFQg")
$runSchedules = Invoke-WebRequest -ContentType "application/json" -Headers $headers "http://localhost:9001/api/v3/schedules" | ConvertFrom-Json
foreach($runScheduleItem in $runSchedules)
{
if ($runScheduleItem.title -eq $schedule) { $runScheduleId = $runScheduleItem.id }
}
if ($runScheduleId -eq "") { throw "Could not find schedule '$schedule'." }
Write-Host "Running the schedule."
# Run the schedule now.
$timestamp = [DateTime]::UtcNow.ToString("ddMMyyyy HHmmss")
Start-Sleep 1
$runNow = Invoke-WebRequest -Method PUT -ContentType "application/json" -Headers $headers "http://localhost:9001/api/v3/schedules/$runScheduleId/runNow"
if ($runNow.StatusCode -ne 200) { throw "Could not run schedule." }
$runNowResponse=$runNow.Content | ConvertFrom-Json
$runId=$runNowResponse.RunId
# Get the result, keep polling every 5 seconds until a new result is returned.
do
{
Start-Sleep -Seconds 5
Write-Host "Polling for run results."
$runResult = Invoke-WebRequest -ContentType "application/json" -Headers $headers "http://localhost:9001/api/v3/run/$runId" | ConvertFrom-Json
}
while ($runResult.Status -ne 'Finished')
Write-Host "Results received."
return $runResult
}
# Function that creates a bug with a specific title in TFS or updates if it already exists.
function CreateOrUpdateBug($title, $description)
{
# Create an authorization header using a Personal Access Token.
$pat = "YOUR PERSONAL ACCESS TOKEN HERE"
$patEncoded = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(":$($pat)"));
$header = @{"Authorization" = "Basic $patEncoded"}
# See if a bug with the same title already exists.
$queryAlreadyExists = '{ "query": "SELECT [System.Id], [System.Title], [System.State] FROM WorkItems ' +
'WHERE [System.WorkItemType] = ''Bug'' AND [System.Title] = ''' + $bugTitle + ''' AND [System.State] <> ''Removed'' ' +
'ORDER BY [System.Id] DESC" }'
$urlAlreadyExists = "https://YOURINSTANCE.visualstudio.com/DefaultCollection/YOURPROJECT/_apis/wit/wiql?api-version=1.0"
$header['Content-Type'] = 'application/json'
$resultAlreadyExists = Invoke-RestMethod -Headers $header -Method Post -Body $queryAlreadyExists $urlAlreadyExists
# If the bug already exists, update it.
if ($resultAlreadyExists.workItems.Count -gt 0)
{
Write-Host "Updating existing bug $($resultAlreadyExists.workItems[0].id)."
# Add a note that this bug was updated or re-opened.
$timestamp = [DateTime]::UtcNow.ToString("dd-MM-yyyy HH:mm:ss")
$bugDescription = "Updated or re-opened by LEAPWORK on $timestamp.
" + $bugDescription
# Update bug.
$queryUpdateBug = '[ { "op" : "replace", "path" : "/fields/System.Title", "value" : "' + $bugTitle + '" }, ' +
'{ "op" : "add", "path" : "/fields/Microsoft.VSTS.TCM.ReproSteps", "value" : "' + $bugDescription + '" }, ' +
'{ "op" : "replace", "path" : "/fields/System.State", "value" : "New" } ]'
$urlUpdateBug = "https://YOURINSTANCE.visualstudio.com/DefaultCollection/_apis/wit/workitems/$($resultAlreadyExists.workItems[0].id)?api-version=1.0"
$header['Content-Type'] = 'application/json-patch+json'
$resultUpdateBug = Invoke-RestMethod -Headers $header -Method Patch -Body $queryUpdateBug $urlUpdateBug
}
else
{
Write-Host "Creating new bug."
# Create new bug.
$queryCreateNewBug = '[ { "op" : "add", "path" : "/fields/System.Title", "value" : "' + $bugTitle + '" }, ' +
'{ "op" : "add", "path" : "/fields/Microsoft.VSTS.TCM.ReproSteps", "value" : "' + $bugDescription + '" }, ' +
'{ "op" : "add", "path" : "/fields/System.State", "value" : "New" } ]'
$urlCreateNewBug = 'https://YOURINSTANCE.visualstudio.com/DefaultCollection/YOURPROJECT/_apis/wit/workitems/$Bug?api-version=1.0'
$header['Content-Type'] = 'application/json-patch+json'
$resultCreateNewBug = Invoke-RestMethod -Headers $header -Method Patch -Body $queryCreateNewBug $urlCreateNewBug
}
}
# Run the LEAPWORK schedule "My Test Schedule" and get the results.
$runResult = RunScheduleAndGetResults("My Test Schedule")
# If there are any failed cases in the results, iterate through them.
if ($runResult.Failed -gt 0)
{
Write-Host "Found $($runResult.Failed) failed case(s)."
$headers = @{}
$headers.Add("AccessKey","bTyGAd0UGL70JFQg")
$runId=$runResult.RunId
$runItemIds = Invoke-WebRequest -ContentType "application/json" -Headers $headers http://localhost:9001/api/v3/run/$runId/runItemIds | ConvertFrom-Json
$rootPath =$runResult.RunFolderPath
foreach ($runItemId in $runItemIds.RunItemIds)
{
$runItems = Invoke-WebRequest -ContentType "application/json" -Headers $headers http://localhost:9001/api/v3/runItems/$runItemId | ConvertFrom-Json
if($runItems.FlowInfo.Status -eq 'Failed')
{
# Create a title for the bug.
$bugTitle = "LEAPWORK: " + $runItems.FlowInfo.FlowTitle
# Create a description that contains the log messages.
$newline = "\r\n";
$keyFrames = Invoke-WebRequest -ContentType "application/json" -Headers $headers http://localhost:9001/api/v3/runItems/$runItemId/keyframes/1 | ConvertFrom-Json
$bugDescription = "Log from LEAPWORK:$newline $newline"
foreach ($keyframe in $keyFrames)
{
if ($keyframe.Level -ge 1)
{
$keyframeTimestamp = get-date($keyframe.Timestamp.LocalDateTime) -Format "dd-MM-yyyy HH:mm:ss"
$bugDescription += "$keyframeTimestamp - $($keyframe.LogMessage) $newline"
}
}
# Add path to video and screenshots.
$mediaPath = Join-Path -Path $rootPath "$($runItems.RunItemId)"
$videoPath = Join-Path -Path $mediaPath "$($runItems.RunItemId).avi"
$videoPath = $videoPath.Replace('\', '\\')
$screenshotsPath = Join-Path -Path $mediaPath "Screenshots"
$screenshotsPath = $screenshotsPath.Replace('\', '\\')
$bugDescription += "$newline Video: $videoPath $newline"
$bugDescription += "$newline Screenshots (if any): $screenshotsPath $newline"
# Create or update bug in TFS.
CreateOrUpdateBug($bugTitle, $bugDescription)
}
}
}
else
{
Write-Host "No failed cases found."
}
The previous example script runs a pre-defined LEAPWORK schedule, polls for the results until they are available, and then loops through all failed cases and creates or updates bugs in VSO, which is Microsoft's Azure cloud solution, as appropriate.
Note: The script contains no error handling or logging mechanisms. It is meant to only demonstrate the core functionality of integrating LEAPWORK with Azure DevOps Server.
After running, cases are created in Azure DevOps Server and can be managed, for instance, through Visual Studio:
The previous script uses PATs to authorize access to Azure DevOps Server. For more information about how to setup PATs and how to perform Azure DevOps REST calls, see the following links:
https://www.visualstudio.com/en-us/docs/integrate/get-started/rest/basics
If you have the Controller installed on your computer, explore the previously mentioned endpoints by going to the following URL: http://localhost:9001/help/index.
Read the full documentation on the LEAPWORK REST API.
If you have any questions, contact priority support on prioritysupport@leapwork.com.