When people mention about WebDriver or Appium, they are talking about software testing tools: libraries and applications that can be used to create, run, and store tests. The framework is the next level up: a tool that can run many tests, perhaps on a schedule, and create meaningful reports of the results. It’s also common for frameworks to store the results of multiple test runs in history and to show trends or historical data.
Some frameworks produce output that can be interpreted by other computer programs, such as a continuous integration (CI) server; others produce dashboards to be interpreted by humans. Some integrate multiple kinds of testing, done by humans or other tool results, into wrap-up posts. Running and interpreting tooling by hand can be a fast way to get started, but any serious test tooling effort running more than a few months will want a framework.
Should you buy a framework, or build one, and if you build it yourself, should you use open-source components? Here's what you need to consider.
Getting started with frameworks
If your team has CI that creates test reports, if the test tooling runs from a single command, if the CI system knows how to interpret the output, and if the CI system provides all the reporting, there may be no need for a framework at all. Michael Larsen, a senior QA engineer at Socialtext, says that at a previous job, his framework was a series of small Unix shell scripts that wrapped, called, and managed code built with open-source components.
This fits the classic definition of framework as a tool that inverts control. Where a classic library, such as Selenium, gets called by test code, the framework is the one that is executing the test code.
Richard Bradshaw, an independent test consultant and an organizer for the Ministry of Testing, which runs TestBash, suggests some possible advanced features for a framework. First, the framework could provide “before and after” hooks, which happen automatically for every test, coded once but ensuring a clean environment between test scenarios.
Bradshaw also suggests tools to enable parallel execution, allowing the framework to run all the tests in half the time, or a fourth or a sixteenth of the time, or even less. To accomplish this, he tends to think in terms of testing architectures, across multiple tools.
The buying option
Commercial tools have two common advantages. First, getting started is fast–incredibly fast if the product runs as a website hosted externally. In that case, setup is a matter of a one-time configuration. For example, if the test code is hosted on GitHub and test servers have been created in the cloud, an online continuous deploy tool could do all the work of a framework, including creating dashboards and running tests. And even if you go with a local install because your company is uncomfortable hosting code on the web or creating the firewall rules to expose the data center, setup is still very fast.
The second advantage of commercial tools is that they provide pretty reports that you can drill into, with historical data, charts, graphs, and multiple sources of test input.
Test case management tools frequently combine the ability to run and port on tests in multiple tools at multiple levels (unit, integration, and end-to-end) with the ability to report on the results of human exploration. The dashboards that result can provide a single unified view, including insights into what kinds of testing find what sorts of defects.
Those sort of capabilities are extremely rare in open-source frameworks, and would take excessive work to build. On the other hand, commercial tools are much more likely to support only a single way of doing things—for example, running Selenium through jUnit.
jUnit is a popular unit test runner, but there are dozens of ways to run Selenium, and a half-dozen supported languages. Even if you use jUnit now, you might not want to in six months or six years—at which point you could find yourself at the mercy of a vendor.
The build option
The great advantage of building a framework yourself is control. With the homebrew automation path, what you need will be exactly what the team builds.
Curtis Pettit, senior test engineer at the digital marketing firm Huge, suggests that the test code should run in the same language that the programmers use, and ideally be edited in the same IDE. His goal is to get the feedback loop from failures into the programmers’ workflow. In that way, instead of the tester running a test program at the end of a two-week sprint and filing a bug, the CI system informs the programmers of the problem within an hour of the commit.
In the event of test failures, production programmers debug the failure, “greening” the test or fixing the production code. Unless the organization uses Microsoft tools exclusively, it is unlikely that a single commercial framework can weave intimately with every programmer’s workflow in a heterogeneous environment—but it might be possible if the commercial framework has a plugin architecture.
The typical problem with homebrew tools is a lack of formal support, especially if the creator of the tool leaves the company. Some open-source tools have commercial support; they may also have an engaged network of enthusiastic supporters. nUnit and jUnit are two unit-testing frameworks that are open source yet have blogs, books, and fan forums available for help.
Bradshaw points out additional advantages of open-source tools. The more popular open-source frameworks have large communities that will have quick turnaround on bug fixes; there is also plenty of documentation available online. Reputation is built in these communities through contribution, leading to a wide array of getting started-tutorials, tips and tricks, usually on the web at no cost.
Bradshaw also points out that commercial tools often have a forced upgrade path, whereas open-source tools that just work can keep working forever. Companies that use the tools can upgrade them, or not, at their leisure.
Larsen says that one company he worked with built a test framework on top of Selenium RC 1.0 in 2006. Now over ten years old, the framework “just works,” but it was built in Perl, which is hard to staff now, and it uses an out-of-date, unsupported version of Selenium. The company is considering rewriting the tool with WebDriver in Node.js, or another newer language, which feels like redundant work.
Count the cost
Kelly Emo, a developer evangelist with Micro Focus, formerly HPE Software, points out that the loudest voices in the room might not best represent your organization's needs.
“With regard to rolling your own, Netflix, Microsoft, and Facebook are doing industry a disservice because they are unique organizations. Most organnizations can’t afford to maintain that level of developers on staff.”
—Kelly Emo
For example, if a test framework results in a 2% productivity improvement and is built by a team of 10 programmers, it would require 500 programmers to see a benefit from the framework. With fewer than 500 programmers, the framework will have a net negative productivity impact. As Ozan Onay puts it, you are not Google, you are not Amazon, you are not LinkedIn.
If your organization creates a checklist of requirements that are impossible—if no tool has everything the organization wants—first count the cost. Then re-examine if those are really requirements or just “desirements.”
Sometimes the “must haves” turn out to be “nice to haves.”
Making the decision
In most cases, you don’t have to make any decision to get started. That is to say, you can do both. Most commercial frameworks have a 30-day trial for free, so you could download one or two while building a minimum viable product (MVP) homebrew framework.
After 20 days, it should be clear whether the commercial tool will work, whether you can budget for it, and whether the homebrew approach has a chance. Bring both options to management or the purchasing authority. If the homebrew approach will not be done in time, explain the consequences of not continuing with the commercial approach.
Keep learning
Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
- Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.
