Chatbots have become a popular way to enhance customer support. There’s no need to go deep into the benefits. Customers don’t have to wait until a human assistant is available to get the answers they need. Meanwhile, companies can increase satisfaction rates, simultaneously reducing costs required for a customer support unit.
But like it always happens with technologies, there’s one “but.” You need to make sure that a chatbot will be a decent alternative for a human assistant – an equivalent for one.
The truth is, a bot can communicate with lots of customers simultaneously, but the value of this communication will depend on a number of tech aspects. In other words, you will need to build a smart chatbot that covers major user requests. And for this, you’ll need a well-thought-out business logic, accurate implementation, and thorough testing.
In this article, I’ll focus on the latter – chatbot testing and quality assurance activities that will bring you closer to releasing a smart and functional chatbot.
The main purpose of a chatbot is to deliver the best user experience and value through conversations. Does it mean that an average user can cover the testing? Unfortunately, no.
On the one hand, you don’t need to have any specific skills to estimate whether a chatbot shows clear answers and offers enough queries. Try out several scenarios and see if the information you get meets your expectations.
On the other hand, the communication flow is one of many aspects that require testing. User interface, response speed, and error handling are among the other things that need a QA inspection. And unlike conversational flow, an average user won’t share insights that would be valuable for quality enhancement.
If you have doubts about the necessity of involving professional QA resources, consider the common chatbot failures. I’ve analyzed some data on this subject to come up with the list of common issues, and it consists of the following points:
If a chatbot is not helpful, it just frustrates customers, affecting your brand’s image and business efficiency. Meanwhile, software testing makes it possible to prevent all the mistakes listed above.
A QA team needs to create a custom testing strategy for a chatbot, just like for any other software product. This article isn’t a definitive guide on how to create such a strategy, but you’ll certainly find some useful hints and tips on what to test and how to improve the quality of an end-product.
A chatbot can be effective only in case it understands the context where it operates. To make this happen, developers need to ‘teach’ an application different word categories and specific terms.
Let’s say, we have a coffee shop chatbot that lets users place orders. You can list options and make them clickable or teach the bot to recognize orders. In the first case, it may be easier for developers to program the functionality, but it may be less convenient for users. Take a look at an example of such flow:
This flow can vary, and you may need to add other stages – hot, cold & iced; with tea, alcohol, or flavor; and so on. The logic will get more complicated as you add an opportunity to order coffee beans, deserts, or other items a customer can normally find in a coffee shop.
The other scenario is more complicated. It deals with direct user requests and extracts the necessary information. For example, a user sees a “Hi! What would you like to order?” message and types the reply. Let it be “A cappuccino with one sugar and a no sugar cold brew for Ben.”
There are several entities here a chatbot should understand:
Developers insert these data at the early stages and build the chatbot logic around it. Reading and extracting the information by a functioning bot is also known as slot filling.
To find out if the bot is familiar with core entities, a QA engineer will need to run domain-specific tests. It is essential to verify that the chatbot understands all the coffee-related lingo customers may use.
Make sure you’ve added a sufficient number of entities and values for each so that the chatbot can understand a request correctly. In a perfect case, you will need to cover both scenarios – one for those who already know and one for those who are choosing what to order.
If a chatbot accepts inputs like email addresses, phone numbers, and zip codes, it is critical for the system to recognize a correct data format.
Feed the chatbot email addresses, phone numbers, and zip codes in different incorrect formats, for example:
Thus, you will learn how the system accepts invalid data and suggest improvements if the response is not user friendly. Here’s an example of such a conversation:
Data validation should be immediate and followed by a corresponding message. The message can show an example of a correct email address, mobile number, etc., or just notify about the mistake. If the input is valid, a user should see a ‘success message’ – proof that the system has received your data and recognized it as valid.
A team cannot foresee all the possible user inputs, especially if there is an input option that allows typing a request. Therefore, it is necessary to investigate the boundaries of the programmed logic. The other task is to make sure the bot has decent responses to the inputs it doesn’t understand.
Run exploratory tests to learn how a chatbot handles non-standard inputs. Such inputs can include:
Customer creativity has no boundaries, especially when they want to mess with a chatbot. You cannot cover all possible scenarios and messages. There are several things to do:
A chatbot is a business tool, not a toy users can test for smart responses. And nevertheless, humanization features have a positive effect on brand image and user experience.
When it comes to swearing, however, it would be reasonable to keep up with the brand’s tone of voice. Make some jokes and respond to offensive queries if you can. Otherwise, just quit strategically. The only thing that is common regardless of the communication strategy is to never swear back. It isn’t very polite.
Chatbots often show buttons with response options, provide links to a brand’s website or other resources, have cards with embedded information or other images. All these elements should be functional – links lead to the correct sources, images optimized and displayed correctly, all the chains of responses connected so that a conversation doesn’t stop abruptly.
It is necessary to check the entire flow looking for broken links, graphic elements, and gaps in logic.
Provide an opportunity for users to leave feedback. Thus, they will report if they find some defects you might have missed. 100% test coverage is a myth, and leaving a couple of little things unnoticed is totally okay.
One more essential thing is to ensure that a chatbot looks properly on different devices. It is a common practice to integrate a bot into a messenger, but that’s not the only option. While messenger design has been already customized for a variety of devices, connecting a chatbot to a website may not be as seamless.
Check how a chatbot window looks on different devices, browsers, and OS versions. Resize a browser window on the computer screen to make sure that the graphics appear undistorted and the layout isn’t broken. Texts should remain readable and not overflow out of the boundaries.
Check the chatbot on widely used devices, browsers, and OS versions before the release. Adjust the chatbot window style and parameters whenever necessary.
It is one of the most obvious aspects of a chatbot to check. UX testing will be based on both business requirements and users’ personal experience. It is essential to make sure that a chatbot fulfills its core purpose, generating correct and accurate responses.
Based on the business requirements, create the scenarios for user queries. The main idea is to check these queries from a point of view of a user who sees your chatbot for the first time. So, start by answering the following questions:
The main rule is to keep a conversation as normal as possible. So here are some basic rules that will help you to achieve this purpose.
#1. A conversation with a chatbot should have a logical flow, much like a real-life conversation with a human. For example, start with a greeting and follow up with a question to start the conversation:
#2. Pay attention to input variations. If you have a text field where users can type their queries, make sure the chatbot understands different requests. Let’s say, you want to order an item from a stationery shop. There are several options to start a conversation:
It may be reasonable to provide a standard set of answers, it depends on the purpose of your chatbot and the variety of items/services available.
#3. Every response should be logically connected to the previous query or question. It is critical that the chatbot provides only relevant and required information. Notify a customer about errors in their queries instantly. Confirm the success message in case the chatbot requests to enter contact data.
#4. There should always be a final message signaling that the conversation is over. It can provide links to some helpful resources, guidelines for further communication, or a goodbye phrase.
#5. Consider an opportunity to connect a user to a human assistant. Whether you want it or not, there are some questions that bots cannot handle. And just like that, provide an opportunity to come back to the beginning or retrieve to the previous step. A chatbot requires clear navigation, just like any other online service.
People using chatbots expect to get accurate responses to their requests and to get them instantly. It means that chatbot performance testing is essential. Therefore, latency – the time a chatbot needs to retrieve the response and display it – should be minimum.
A QA engineer should check different queries that cover small and large dialogues and feature different types of content. It should take the same time for each response to appear on the screen, regardless of its place in a user journey and featured content.
The delay that lasts more than a couple of seconds causes frustration, and a user is likely to abandon a chatbot. The delay may depend on the response’s content, but the recommended response time is up to two seconds.
As a rule, a chatbot doesn’t function on its own – it becomes an integral part of the company’s digital ecosystem. That is, you’ll eventually need to connect it to the official website, a social network account, messenger, application, or another digital medium.
If you use bot building software, you are likely to find an API that allows connecting a chatbot to your system and run API tests. If you use a custom solution, it is essential to run integration tests to check how a bot and other system features behave when connected.
Pay attention to both tech and design aspects of your chatbot. It is a part of your digital system, and it should integrate seamlessly. All the functional and performance aspects mentioned above should meet the quality standards after a chatbot is connected to the website or other platform. The design should be consistent with other media and brand image in general. The latter aspect falls under UX testing, but the inconsistency may become apparent at this stage.
Last but not least, avoid errors in responses – spelling, grammatical, lexical, any errors! In most cases, they won’t have a catastrophic effect, but attentive users can get skeptical about the level of services you deliver. Long story short, little things matter, and that’s why texts should be error-free.
If a website is multilingual, it makes sense to create a chatbot that is multilingual as well. Keep in mind that localization encompasses other aspects as well – time zones, currencies, metric systems, and more.
Entrust localization testing to someone good at editing texts. If a QA specialist isn’t confident in their proofreading skills, assign this task to another person on the team. Besides, a fresh perspective is always helpful.
So, check the responses for spelling, lexical, and grammatical errors, punctuation, and cohesion. Make sure the formatting is correct, too. Whenever you have things listed, use bullets or numbers. Keep sentences short and divide information into paragraphs in case a message is long.
It would be great for a chatbot to recognize the user’s time zone and greet them accordingly. As for the language, the system should be able to access the corresponding device settings or start the conversation flow by specifying these details. If currency and other localization details are important, the chatbot should specify those via questions, too.
Testing a chatbot is not an easy task, and it requires the participation of a professional QA company. Besides, the work on a software solution doesn’t end with its release – a chatbot will require maintenance and, most likely, improvements. After the launch, a company starts to receive user feedback that always comes with useful insights for quality enhancement.
Don’t forget that a chatbot can learn from interactions with users. A detailed analysis of user requests, bot responses, missing queries, etc. helps to gather data you can use for enhancing user experience. And each new bot version will require a set of tests to verify it works just fine 😉
Quality control is obsolete. The spread of Agile, DevOps, and shift-left approach has pushed traditional…
Be honest, if your phone disappeared right now, your world would be in shambles. Data…
Teams have a love-hate relationship with Android. It’s highly customizable and has an incredibly vast…
Apple applications are easy to test. Compared to Android, that is. But when it comes…
Result-driven QA isn’t always about planning and strategizing. Sometimes, the best thing for your product…
A flimsy UI doesn’t lead to customer frustration, negative reviews, and high churn. When people…
