When you're building a chatbot or virtual assistant, the quality of the conversation should be the most important consideration. A chatbot should adopt features and characteristics that deliver a high-caliber conversational experience.
You can find resources that describe the many advantages of using chatbots for business, but the conversational aspect never seems to receive the attention it should. Run a quick web search for "advantages" or "benefits" of using a chatbot, and here is what you'll find:
- It's available 24/7.
- It scales.
- It helps you save money.
- It automates your work.
- It cuts down on human error.
While all of those attributes are very useful and are definitely part of what a bot can help a business with, they are not unique to bots.
Understanding that human-to-human interaction is the most effective way to communicate and get things done is key to building successful chatbots and virtual assistants.
What does it mean to be 'conversational'?
What exactly does it mean for a bot to be conversational?
In their paper "Perspectives for Evaluating Conversational AI," researchers Mahipal Jadeja and Neelanshi Varia identify the characteristics that people generally expect of a human assistant; these features are what people expect to see in a high-quality virtual assistant or chatbot.
- Knowledge about the user—This includes things such as the time of the day the user typically prefers to set meetings and whether she is more of a tea or a coffee person.
- Problem solving and influencing skills—A good personal assistant knows his boss (the user), understands her mood, and can come up with appropriate words to avoid conflict and offer some relief during a stressful day/season.
- Security and trust—Confidentiality and a sense of mutual trust are maintained at all costs.
- Efficient organization—The personal assistant should have relevant files, documents, and so on, all well organized and readily available.
- An understanding of the job—The personal assistant should understand the rules and methodologies the company has adopted.
This is a great place to start. Knowing that the above features are what's expected of a good personal assistant, you can build your virtual assistant while trying to perform as highly as possible in these areas. (Another good resource is the book Designing Bots: Creating Conversational Experiences.)
You can also look at a list of skills that a virtual assistant using natural language should have. You can watch this video to see how Gillian McCann, head of cloud engineering and AI at natural-language platform provider WorkGrid, defined a "good" natural-language assistant at the 2019 Amazon re:Invent conference. (McCann starts speaking at around 34 minutes into the video.)
According to McCann, a conversational assistant should be able to:
- Perform tasks, including specific actions such as scheduling a meeting or creating a ticket. These are likely to be API calls to third-party systems.
- Provide intelligence. This means personalization, such as offering personalized answers based on where a user is located. It also means memory—that is, the ability to provide and retain context about the user.
- Provide entertainment. The conversation should feel unique, engaging, and interesting.
- Have knowledge. This is the most important element. Ultimately, your chatbot or virtual assistant should be able to answer specific questions with clear, conversational answers.
On the last item, McCann later offers a good mantra to help make your chatbots knowledgeable in a conversational fashion. She calls it "answers, not links." For a regular web search it is acceptable to respond with web links, but that's not the case when it comes to a conversation.
When you ask someone a question, you expect a reply that involves at least a few words! Of course, you could also provide a web link as a plus, but your user will expect more than that.
Measuring your bot’s conversational skills
So far, you have learned about some of the characteristics of a human personal assistant, as well as the expected capabilities of a good-quality natural-language assistant.
But how do you know if your bot is both conversational and effective? Researchers Jadeja and Varia suggest a few universal metrics that everyone can adopt.
Compare your bot's interactions with those of a live human
This metric, called "user perspective," asks you to try to execute the same task that you built your chatbot to do, but with the help of a human assistant instead of the chatbot. When running this metric, observe the quality of the interaction between user and assistant.
This is the most critical evaluation metric you can run, but it's also the most expensive, because it involves relatively specialized humans in the process. The metric is crucial because it gives you a better understanding of the user's expectations, lets you establish trust between the user and the virtual assistant, and helps you to understand the tactics and methodologies adopted by the user to get the job done.
Measure the information retrieval
Here, you want to evaluate whether a bot is able to find and display the information the user had requested. This is called an information retrieval (IR) perspective.
While this metric is very important, it doesn't give you the full picture. For example, a bot may show a list of products the user is looking to buy, but still miss the mark because of some nuance in the way the user asked for it—something that a human assistant would have picked up on—that indicated an interest in a specific variation of those products.
This metric also isn't particularly helpful when it comes to the way the virtual assistant interacts with the user. In other words, this metric may prove that you have a sophisticated search engine behind the scenes that can find anything the user asks for, but it says very little about the way the information is presented back to the user.
Nevertheless, as shown in the features and skills listed above, knowledge (the ability to answer the user's requests with useful results) is essential to a good virtual assistant.
Measure linguistic properties
H.P. Grice, in his book Studies in the Way of Words, outlines the four principles required to achieve maximum cooperation between people having a conversation. These are explained in more detail in the chapter titled "Logic and Conversation," printed in full here (PDF).
- Quality: What is being said should be true and evidence-based.
- Quantity: The amount of information shared should not be too much or too little.
- Relation: The response must be related to the conversation.
- Manner: There should be no ambiguity, and the interaction should be direct and straightforward.
This metric requires judgment by a linguistics expert. As Jadeja and Varia put it, "Who will decide whether the conversational AI's response is related to the topic or not?" There's no standard way to determine that. Still, looking at Grice's four principals can be very effective for improving the level of trust between the user and the virtual assistant.
Run a standard AI measurement
The most well-known AI measurement is the Turing test. The drawback to this approach is that it doesn't give you any recommendations for how to improve your bot's conversational skills.
It may give you a one-off assessment as to how well or badly your conversational AI system is doing, but it won’t give you any direction on how to make it better. This is a great starting point though, so don't discount it.
How to get started
The above, in a nutshell, are the characteristics, features, and skills your bot should display. Some of these are things that you would expect of a human assistant, while others are more specific to virtual assistants (e.g., being able to call an API to provide answers).
By combining these features you'll make a great chatbot that gives converses with your users a the highest level of quality.
Also, try running the universal metrics described above to determine how well you're doing. These measurements will help you discover whether your chatbot is truly conversational. This is what will set your virtual assistant apart, since it allows you to deliver an experience that goes far beyond what the simple web pages or online forms that your competitors may be using can provide.
Keep learning
Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
- Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.
