Customers frequently request for quality assurance (QA) teams to share their tests. Testing with customers is a mixed bag, and whether it's feasible depends on how the tests are written. Additionally, sharing negative testing scenarios or data verification tests may create more problems than it solves. The important consideration may be to define which tests are shared first.
Define what type of tests to share and review those tests to ensure they're ready for customer use. Many times development management releases QA tests to customers before reviewing them. Resist this temptation—it's hazardous. Take the time to review tests to verify that the tests don't communicate information the business doesn't want shared with end users, including confidential data and or proprietary information. This is important for both the business and the customer. If a customer gets tests cases that they either don't understand or that don't seem to test from their point of view, it generates a negative response to the application. Don't shortcut responding to the customer's needs by sharing QA tests that aren't ready or able to be shared.
Development management must also truly understand that QA testing is different than customer testing. Yes, QA does test as much as possible, but other things are involved as well. QA tests often involve changing configuration settings that typical users don't have access to. Additionally, QA tests may include data setup steps that customers don't need or may find confusing. Also, QA tests may include SQL scripts or development code manipulations that don't make sense from a customer's point of view.
Before you release QA tests to customers, determine what the customer's goal is and which types of tests will help them reach it. Check the format the tests are written in, because some tests are written with extra elements that may not make sense to an end user. Finally, review what will be released and verify that it's appropriate to release it.
Determine the goal
Determine the goal of sharing the tests. For example, does the customer need help testing? Are they looking for testing direction or to get an idea of what you've tested, or are they actually using the test to meet an obligation or a regulatory requirement?
In any case, verify what their goal is before you share any tests. Make sure the QA team is involved in identifying appropriate tests and then separating them so they can be scrubbed of any sensitive, confidential, or proprietary data. Consider having the QA team consult with the customer on what their testing objectives are. The business may be better served by having a QA team member work with the client to develop custom tests.
Another option may be organizing shared testing sessions with customers, similar to what has now become the old-fashioned method of acceptance testing. Acceptance testing sessions can be extremely valuable for customers and the business. Sessions can be as short as a day or as long as a week and can occur online or within a shared space. Bringing participants into a shared space is useful to gauge their reactions and responses more accurately. During the sessions, the QA or user experience professional takes notes on where users run into issues, what they are, and how frustrated or creative they get when trying to get around the problem. Testing sessions can be a valuable source of information for application development and allow for feedback on the application prior to an official release.
Check the format
It's important to read through any shared tests and verify that the format is readable to users outside the application's QA department. Just because the tests are being sent to customers doesn't mean they'll be able to understand them. For example, if QA's write tests include keywords for automation, coding jargon, or abbreviations, it may be impossible for a customer to follow the test case.
For example, against my wishes, a former QA manager released nearly 100 tests to an important development partner customer. I have no issue sharing tests and getting feedback or sharing tests and having someone tell me I'm missing something. However, before you share anything, review, review, and review. The customer never used the tests because the tests didn't make sense to them. Why? The tests that were sent were literally written in code, with keywords. If you're not used to reading information that way, it's difficult to decipher what the actual end user's steps are. The customer came away with the impression that we didn't understand how they used the application, and they had no idea what we were testing. It wasn't a positive experience for them and it didn't help an already faltering relationship.
If necessary, have a QA resource clean up the tests and write them in a user story format so they're readable to people outside the QA group before sharing.
Remember the customer
Another plug for reviewing tests prior to sharing them: remember the customer. The customer's impression of the business determines how they view the product and its quality. Don't share bad, invalid, or poorly written test cases. It simply makes you look bad. The same basic principle applies to sharing tests that are too technical, since it'll seem like you only know how to test technically and not from the customer viewpoint.
The important point is to budget time to review the tests before you share them. It's critical to remove confidential or proprietary information before sharing. After all, you can't take it back once it has been made available. Also, remove data verification steps or SQL queries, because clients don't have access to that level of the system. Keep the tests shared based on the end user interface only and don't require access to development code or database tables. Think first, review, and then share if the test meets the expectations and needs of both the business and the customer.
Keep learning
Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
- Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.
