Studio 8.1 groups all executions into one report and makes it much easier to find inconsistent results or false positives.Īdditionally, we have enhanced our test result third-party integrations with tools like TestOps and qTest so the report shows only one final set of results rather than multiple sets of results for each test case. In this release of Katalon Studio, we have improved the logic of our reporting mechanisms so they are more intelligent and intuitive to provide better insights on test results.
The latter is by far the most popular option among Katalon Studio Enterprise users.Īt Katalon we understand how important it is for developers to be able to make data-driven decisions. Katalon Studio users have multiple options to tackle test flakiness: they can retry all Test Cases when the Test Suite finishes retry only failed Test Cases when the Test Suite finishes, or retry a failed Test Case immediately after its first failure.
Edit and improve the Test Case accordingly.To tackle test flakiness, developers rely on the following steps: The unreliable nature of test results yielded by flaky tests is a never-ending war and it represents one of the most important challenges in the test automation industry because, without accurate results, DevOps teams can’t make accurate decisions about their product releases. Test flakiness is caused by issues with new code, external factors like faulty infrastructure setup or network glitches, or an issue with the test itself. Improved retry failed executions immediately to spot flaky testsĪ flaky test is a test that will pass or fail regardless of the changes done to the underlying code. Studio 8.1 delivers features that aim to increase the resource and cost-efficiency of your automation cycle, provide better integration with Azure Test Plans, and enhance mechanisms to tackle test flakiness. Read on to learn more about what’s included in this release. This release comes jam-packed with a broad set of new features as well as dozens of improvements and fixes to continue providing you with a top-notch testing experience. Īs of this writing the most current version of testNG is 6.9.10 and its default configfailurepolicy (what happens when this attribute is omitted from your testng.xml) is set to skip.Hello Katalians! Time flies when you’re busy creating awesome products! Back in May, we delivered Katalon Studio 8, which included features to provide better scalability options and a better user experience.Īnd because we are not in the business of keeping you waiting for the latest and greatest, today, we are excited to announce the release of Katalon Studio 8.1 (KS8.1). However if you wish to have the execution continue on for some reason you can change this to configfailurepolicy="continue". To force the setting of skip add the configfailurepolicy="skip" to the the suite tag of your testng.xml.
#Katalon studio failure handling driver#
In general you likely want your tests to skip because there is no need to attempt to run them if the driver has failed to initialize, to do so would only lead to additional failures and waste additional execution time. When set to skip the methods are skipped, when set to continue the methods attempt to run anyways. What this controls is whether the test methods should attempt to be ran regardless of a configuration failure.
The default behavior of the configFailurePolicy setting has changed, purposely or not, from the behavior of "continue" to the behavior of "skip" over time. If you wish to have a different set of before listeners which perform different behaviors based on the group name then you shouldn't use alwaysRun.