If you're thinking about building your own test automation framework, think again. In most cases, you'd do better to consider one or more of the open-source options available.
That's because, in general, a framework is a set of best practices, assumptions, common tools, and libraries that you can use across teams. You simply don't need to build one that's unique to your dev environment. A framework will help make your test automation code reusable, maintainable, and stable—and save your business from costly defects. After all, even minor bugs can lead to big problems.
Teams that take these benefits to the extreme by building their own elaborate automation frameworks from scratch drive me crazy. That's because they could have easily made use of existing open-source tools and libraries that would have met their needs without writing any code—and, in most cases, with better results.
While using open-source tools is usually a better option than building your own framework from scratch, I'm not saying open-source test automation tools are all you need. Once you understand how to choose the right test automation tool for each role in your organization, you may end up with a mix of commercial and open-source options.
Here are the top open-source testing frameworks, and how to evaluate them.
Test beyond the GUI
Another area most folks forget to include when looking for a test automation framework is the ability to test APIs. I'm a big believer in testing functionality beyond the user interface (UI) whenever possible. And since integration/API tests bypass the UI, they tend to be quicker and much more reliable than GUI tests.
That's why it's important to have a test suite that tests several different layers of your application. For that, you need tools that can test both the UI and non-UI components of your application.
So before you fall into the "build your own framework" trap, be sure to check out these nine open-source test automation frameworks.
[ Become the full-stack automation engineer you were always meant to be. Sign up for Joe Colantonio’s weekly Adventures in Automation email newsletter. ]
Serenity
If you need a Java-based framework that integrates with behavior-driven development (BDD) tools such as Cucumber and JBehave, Serenity BDD (formerly Thucydides) might be the tool for you. It's designed to make writing automated acceptance and regression tests easier.
It also lets you keep your test scenarios at a high level while accommodating lower-level implementation details in your reports.
Serenity acts as a wrapper on top of Selenium WebDriver and BDD tools. It makes writing BDD and Selenium tests easier because it abstracts away much of the boilerplate code you sometimes need to write to get started.
Serenity also offers plenty of built-in functionality, such as handling WebDriver management, managing state between steps, taking screenshots, running tests in parallel, facilitating Jira integration, and more—all without having to write a single line of code.
Provides detailed reports
Serenity creates highly detailed reports. Out of the box, it creates living documentation that you can use not only to view your Selenium BDD test results, but also to document your application.
It also has an extra annotation method, called @Step. When used correctly, @Step gives you an extra level of abstraction that makes your tests more reusable and maintainable.
Serenity recently added integration with REST Assured, the popular open-source Java library for testing RESTful APIs. It creates another wrapper around REST Assured that makes Serenity Selenium tests flow seamlessly. And Serenity's awesome reporting feature also includes REST information, which means you can easily see and validate all requests and response details in your test reports without having to explicitly add any extra logging code.
Serenity takes care of many things you would normally have to code from scratch if you were creating your own BDD framework. It's one of the better open-source frameworks available, although I might be a bit biased: It’s the tool I use for my own day-to-day test automation efforts.
Serenity has also created a JavaScript version called SerenityJS. Since many front-end developers are also responsible for testing, a JavaScript version of an automated test tool is very helpful.
Cypress
Compared to most of the other tools on this list, Cypress is a more developer-centric test automation framework that focuses on making test-driven development (TDD) a reality for developers. One of its design principles was to be able to package and bundle everything together to make the entire end-to-end testing experience pleasant and simple.
It has a different architecture than Selenium; while Selenium WebDriver runs remotely outside the browser, Cypress runs inside of it.
This approach delivers more consistent results, since Cypress can understand everything happening inside and outside the browser. It also gives you native access to every object without requiring you to deal with object serialization or over-the-wire protocols. Basically, you're pulling your application into Cypress.
As a result, it can synchronously notify you of every single thing that happens inside the browser so that you have native access to every distributed object model (DOM) element.
Cypress also makes it easy to simply drop a debugger into your application, which in turn makes it easier to use the developer tools while you're developing.
Robot Framework
If you want to use Python for test automation, you can't go wrong with the Robot Framework. This mature product, created for testers, uses a keyword-driven approach to make tests readable and easy to create. And it includes many test libraries and other tools you can use.
The Robot Framework is Python-based, but you can also use Jython (Java) or IronPython (.NET).
While the Selenium WebDriver library might be the most used external test library, Robot Framework can test things other than websites, such as FTP, MongoDB, Android, and Appium. In addition to all this open-source awesomeness, it has a lot of APIs that help make it as extensible as possible.
The keyword approach used by Robot Framework is great for testers who are already familiar with other vendor-based, keyword-driven test tools, making the transition to open source much easier for them.
If your team is mostly made up of testers, Robot Framework is a great option for your automation framework.
TestProject.io
Are you craving an automation tool that is completely free and also has a robust community built into and around it?
If so, you should definitely check out TestProject.
At the time of this writing, Java and C# implementations of the SDK are available. I've been told that support for Python, JavaScript, and other programming languages is currently in the works.
TestProject's framework was created to allow more testers and organizations to benefit from the two primary open-source tools for automation: Selenium and Appium.
What's unique about TestProject is the add-ons, which allow testers around the globe to use functionality that other testers are sharing in TestProject. Think of them as test automation building blocks that can be reused by others.
It also comes with other features like a test recorder, global automation grid, and much more.
If your teams are made up of mostly QA/testers who don't program, TestProject.io is a must-try.
Sahi
Developed for testing web applications, Sahi offers both an open-source and a pro version. It works as a proxy server that you can use within a browser. From the Sahi dashboard, you can launch the browser you want to test.
The Sahi controller guides you through recording and capturing the interaction you want to perform against the application. When you hover over any element in the Sahi browser, for instance, it shows you all of the actions you can perform against that element.
You can also use the controller to play back your script. Sahi's record and playback functionality makes it easy to get started with test automation of simple HTML applications.
But beware of relying on record and playback; this approach has proved to be a bad practice for creating maintainable, robust automation tests. It's okay for helping create an initial test, but you'll need to customize it to make it reliable and maintainable.
I've heard quite a bit about Sahi, but my first impression of it is that it's not as powerful as some of the other options. That said, lots of folks rave about the Sahi pro version.
Galen Framework
If your automation efforts are focused on user experience design (UX) or layout testing, Galen Framework might be a perfect fit for your needs.
Created specifically for UX testing, Galen has a special syntax and rules you can use to describe and verify your application's layout. It also lets you specify your browser size, then run tests against your application to verify layout specifications. Galen's tests also generate detailed HTML reports with screenshots, and the tool includes a visual image comparison with a cool heat map feature.
You can write tests in a simple text file using the Galen syntax, JavaScript, or Java.
Gauge
Gauge is one of the newest open-source test automation tools for Mac, Windows, and Linux, developed by the folks at ThoughtWorks—the same company that created Selenium.
Practitioners of TDD and BDD will appreciate Gauge's focus on creating living and executable documentation.
Gauge automation tests, called "specs," are written in a simple markdown language syntax using Java, Ruby, and C# within developer IDEs such as Eclipse and Visual Studio. You can also extend Gauge’s functionality with its support for plugins.
Gauge is pretty new—it's still in beta—but if your team is into cutting-edge test development, you might want to give it a try.
WebDriverIO
Are you looking for an automation framework that leverages Selenium and allows you to program in JavaScript?
WebdriverIO gives you a full-fledged framework out of the box. It doesn't require you to implement anything from scratch, as you would with straight Selenium.
It comes with everything you need to build scalable and sustainable test suites and contains extra features not found in the straight Selenium version. If your team is made up of JavaScript developers and testers who can code, this framework will likely make you happy.
OpenTest
OpenTest is a framework I just learned about recently that can be used to test web and mobile apps and APIs.
If you look at the architecture of OpenTest, you'll find that there are very few test automation scenarios that are not supported. As long as a device can run Java, that device can be used as part of your distributed test—and that happens seamlessly.
One of the claims of OpenTest is that it requires no coding skills. This is achieved using a combination of actions (keywords) written in English and stored in a YAML file. This approach is similar to the way Gauge works.
If cross-platform automation is vital to your team, then OpenTest might be a framework you'll want to try out.
Citrus Framework
Citrus is an open-source framework that will help you automate integration tests for pretty much any messaging protocol or data format. If you have a test flow that requires that your application interact with other services, then APIs and components using functional test tools such as Selenium can't help.
If you are using any kind of messaging transport such as HTTP, REST, SOAP, or JMS, Citrus is the framework of choice to test messaging integration.
Citrus also integrates with Selenium. This is handy if you need to interact with a user interface and then verify a back-end process. For example, say you had to click on a "send email" button and verify on the back end that the email was received. Citrus can receive this email or the JMS communication triggered by the UI, and verify the back-end results, all in one test.
In this way, Citrus gives you true end-to-end test coverage that you can't get with just a GUI test tool.
Karate-DSL
If you're already using Cucumber and need to also test a REST API, Karate-DSL might be a perfect option for you. It's been around for almost a year now, and more and more folks are adopting it for their API testing needs.
Since Karate's REST test tool is built on top of Cucumber-JVM, you can run your tests, view reports, and leverage any other Cucumber functionality, just as with any standard Java project.
Time to choose!
Before you write that first line of code to create your own framework, make sure there isn't an existing library or framework you can leverage. Stop wasting your time reinventing the wheel; check out these automation frameworks first. And for more detailed discussions, visit my automation testing blog.
These tools are my picks, but they're just a small sampling of the open-source test automation frameworks out there. What are your favorites? Let me know what you think I should add next time—and why—in the comments below.
This story was updated in December 2019.
Keep learning
Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
- Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.
