Do you have a Windows desktop application that needs to be tested to verify if all the different features work seamlessly and according to documentation? We suggest you use test automation—or at least try it in combination with manual testing.

Test automation is one of the most efficient ways to accelerate the testing process while ensuring software quality is high. Not only does it reduce the time needed for testing by automating repetitive tasks, but it also brings down testing costs. This is because automated tests can be run over and over again at no additional cost.

To get the most out of automated testing, however, using automation testing tools is essential. But not just any tools, open-source automation tools. The reason for this is because open-source tools are freely available to the public. This means that anyone can view, modify, or enhance the source code. Namely, these types of automation tools allow QA engineers to perform various tasks more efficiently and ensure testing runs more smoothly. By using automation testing tools, you can enter test data into the system under test, analyze test results, perform various types of validation, and take screenshots of the system at some point of validation. You can also generate detailed test reports, log files, and error message files.

While automated testing cannot completely replace manual testing, it is the better choice in various cases, like when you have repetitive tasks, need to run test cases on multiple devices simultaneously, or want to get more accurate and consistent results.

But how do you know which open-source automation tool is right for your project?

In this article, we will look at several different open-source automation tools for testing Windows desktop apps and evaluate each using predefined criteria. Let’s jump right in.

Open-source automation testing tools for Windows desktop apps

We will look at and evaluate three open-source automation testing tools: SikuliX, Winium, and WinAppDriver.

SikuliX

SikuliX is an open-source tool that can automate everything that can be seen on the desktop screen running on Windows, Mac or Linux/Unix operating systems. SikuliX uses image recognition technology provided by OpenCV to identify GUI components.

This particular tool has a visual approach to GUI automation using screenshots. It allows you to take a screenshot of GUI elements—toolbar, icon, or dialog box—and ask the system/machine for help to automate the handling of elements and their actions using screenshots instead of element names. SikuliX also has an integrated visual scripting API that helps automate GUI iterations using image models for mouse and keyboard events/actions. On top of that, SikuliX has optical character recognition (OCR) features that can be used to search for text in images.

Having used SikuliX for testing purposes, I can say that all of the technology described above is convenient for automating both desktop and web applications when the internal elements of the GUI or the source code of its application or web page are not easily accessible.

From my personal experience I can say that even less experienced testers can easily and quickly learn how to work with SikuliX, as research shows that finding elements and writing code with screenshots is easier and faster than with keywords, for example.

SikuliX supports Python as the primary scripting language, but I have also used Ruby for my test scripts and it worked fine.

SikuliX user interface description and explanation. Source: Software Test Academy

Winium

Winium is an open-source automation tool built on the Selenium library to interact with Windows desktop applications. It works with all WebDriver compatible programming languages, including Java, Objective-C, JavaScript with Node.js, PHP, Python, Ruby, C# and so on.

This tool only supports and helps test different types of applications that have been developed in the following environments: WPF and WinForms.

Based on my experience, working with Winium is quite quick and easy if the tester has prior experience and knowledge of working with Selenium. For inspection purposes, I used Inspect.exe (available in Windows by default) to identify elements in a Windows application, although you can use UISpy.exe in Winium for that purpose. There are no built-in UI inspection tools, so third-party tools need to be used.

Winium can be used with any front-end test library. I used it with Cucumber and Gherkin, but it is also possible to use it with Gauge to read test automation steps.

Despite a number of benefits, active monitoring, maintenance, and debugging are key to open-source tools. It’s important to note, however, that the developers of Winium have not made any changes or updated it since 2016. So if you want to use this tool, you have to take into account that errors will not be fixed and the tool will always remain exactly as it is now.

WinAppDriver

WinAppDriver is an open-source testing tool developed by Microsoft as an open source project. It is based on the popular WebDriver protocol and Selenium. Therefore, WinAppDriver is a Selenium-like automation tool. In my opinion, WinAppDriver combines the best of two worlds. On the one hand, it incorporates most of the now obsolete CodedUI technology, while on the other, it combines this technology with the flexibility, ease of use, and variability of Selenium.

Like Selenium, WinAppDriver is a set of libraries that can be integrated into any Test Runner that supports it. For example, WinAppDriver scripts can be developed and executed with MSTest from Visual Studio.

The tool supports both 32-bit and 64-bit applications developed in one of the following environments: WinForms, WPF, UWP, and Microsoft Foundation Class (Win32).

WinAppDriver does not have its own scripting language, but it can write scripts for automation in the most popular programming languages, such as C#, Java, Python, and Ruby.

Like Selenium, WinAppDriver elements are found by identifying properties like class, ID, name, XPath, and so on. WinAppDriver also does not have a built-in element inspection tool, but you can use third-party tools such as Inspect.exe.

The WinAppDriver documentation is fairly easy to find, but from my experience it is very unstructured and non-detailed. It does not contain any video/audio tutorials and there is only one step-by-step tutorial explaining how to install WinAppDriver.

Evaluation criteria for open-source automation tools

Before we dive into evaluating the different open-source automation tools available, we need to discuss the evaluation criteria that we will be using. The list of evaluation criteria that you can see below is unique and can only be viewed within this work. Criteria were selected in a way that they could be evaluated as objectively as possible based on theoretical information and practical usage of the open-source automation tool by the author. The criteria should cover as many different aspects of the tool as possible: different types of support (programming language, applications, debugging, test type, database), built-in tools, common functionality and some built-in features, integrations, documentation, online support and so on.

Each of the listed criteria will be evaluated according to a 5-point system:

5 – Excellent (the tool fully meets the specified criteria)
4 – Good
3 – Average
2 – Poor
1 – Very poor (the tool does not meet the specified criteria at all)

Additionally, at the end of the evaluation of each tool, the average score (the total number of points divided by the number of criteria) will be calculated. This average will also be considered as the final score for that particular tool. All the criteria against which the tools will be evaluated and compared are outlined in the list below:

• Programming language support
• Built-in UI inspection tool
• Required coding skills
• Built-in ALM integrations
• Available documentation
• Ease of installation and use of the tool
• Execution of test scripts
• Record and playback function
• User interface
• Tool training process
• Online support of the tool
• Debugging support
• Creating and storing reports, logs
• Different types of testing support
• Database application support
• Data-driven testing
• The evolution of the tool over the years
• Code/script reusability

Rating the automation tools based on predefined criteria

Based on the information I read about each of the tools, as well as the practical application of the tools—which I specifically performed to be able to evaluate each tool more objectively by creating and running a few test scripts on each one—I reached some general conclusions and used the predefined criteria to evaluate each tool accordingly. The results can be seen in the table below.

Criteria/Automation tool	SikuliX	Winium	WinAppDriver
Programming language support	4	5	5
Built-in UI inspection tool	4.5	2	2
Required coding skills	4	3	3
Built-in ALM integrations	0	1	1
Available documentation	2.5	2	2
Ease of installation and use	5	2	4
Execution of test scripts	3.5	4	4
Record & Playback function	5	3.5	1
User Interface	3.5	2	2
Tool training process	3.5	2.5	2
Online support of the tool	1	0	0
Debugging support	5	2.5	3
Creating/storing reports/logs	4.5	2	1.5
Different types of testing support	3	3	2
Database application support	2.5	4	0
Data-driven testing	3	3.5	0
Evolution of the tool over the years	2.5	0	1
Code/scripts reusability	4	2	2
Final score	3.4	2.5	2

Key takeaways

In this article, we looked at three different open-source automation tools for Windows desktop applications and put them to the test, evaluating them according to 18 different criteria.

The comparison and evaluation of each tool was carried out both on the basis of the available documentation about the tool, as well as the practical application of each tool for a specific test case that was created and performed by me.

Based on my findings, the best open-source automation tool for Windows desktop applications—out of the three we evaluated—is SikuliX, which has an average score of 3.4. This is a relatively high score, considering it is completely free. SikuliX came on top in terms of built-in UI inspection tool, ease of installation and use, and script reusability, among other criteria, of course.

So if you are developing a Windows desktop application and you currently do not have the opportunity to invest in paid test automation tools, then SikuliX is definitely your go-to open-source automation tool when looking for a tool that is easy to use and implement into your testing processes.

Are you interested in introducing automated testing into your current QA processes? Our experienced and highly skilled automation engineers can help boost your testing efforts and increase efficiency by implementing various automation tools and working together with your team. Whether you have a mobile, web, or desktop application—we can help! Drop us a line and let’s discuss your project.

Open-Source Test Automation Tools for Windows Desktop Apps