Table of Contents
- The Importance of Measuring Chatbot Effectiveness
- Examples of chatbot metrics
- response rate
- resolution rate
- Start count
- Number of transitions to the site
- Satisfaction with response
- Number of inquiries for manned support
- Procedures for measuring chatbot effectiveness using evaluation metrics
- Step 1. Clarify the purpose of chatbot operation
- Step2.Determine the evaluation indicators necessary for effect measurement
- Step3. Set target values for evaluation indicators
- Step4.Actual data collection and measurement
- Step5. Compare the data before and after introducing the chatbot
- Step6. Consider measures for improvement
- Evaluation metrics are important when introducing a chatbot
When a company introduces a chatbot, it is important to set and analyze “evaluation indicators” to achieve its purpose and goals. If you proceed with the introduction of chatbots without establishing evaluation indicators, you may end up with a service that deviates from your goals, so be careful. Also, it is important not only to create an evaluation index, but also to follow the numerical value in the subsequent effect measurement. Let’s make a hypothesis while chasing the numbers and try to improve the chatbot.
The Importance of Measuring Chatbot Effectiveness
First of all, I will introduce the necessity of measuring the effectiveness of the introduced chatbot.
Effectiveness measurement is an important process that allows you to check the degree of achievement of goals and review policies. Companies that introduce chatbots aim to solve internal issues such as improving operational efficiency and reducing costs.
Therefore, when introducing a chatbot, it is important to measure the effect and verify whether the introduction can achieve the purpose.
Also, if a chatbot is introduced as a means of solving problems, be sure to measure the effectiveness to evaluate whether it is worth the introduction cost. Furthermore, by making improvements based on the results of effect measurement, you will be able to use the tool more effectively. Especially in contact centers, it is important to regularly measure effectiveness in order to meet user needs.
Examples of chatbot metrics
Chatbot evaluation metrics include response rate and resolution rate. I will explain in detail the characteristics and how to output each value.
response rate
Response rate is how well the chatbot is able to respond to user questions. Compare the number of answers provided by the chatbot with the number of questions asked by the user to get a number.
For example, if a chatbot provides 80 answers to 100 questions, the response rate is 80%. The higher the response rate, the more effectively the chatbot answers user questions.
resolution rate
Resolution rate is the percentage of successful resolutions of user issues and requests. Calculate the percentage of problems that users asked for a solution using the chatbot that were actually solved and give a numerical value.
For example, if a chatbot solves 90 out of 100 problems, the resolution rate is 90%. The higher the resolution rate, the better the features and capabilities of the chatbot.
Start count
The number of activations is the number of times the chatbot was activated. Measure the number of times the chatbot is activated and used within a specific period of time and output a numerical value. For example, if the chatbot is activated 100 times in one day, the number of activations will be 100. The higher the number of launches, the more frequently used by users.
Number of transitions to the site
Shows the number of times users navigated to your website through a chatbot. Based on the guidance of the chatbot and the information provided by the user, it measures the number of times the user accesses other pages and outputs a numerical value. The larger the number of transitions to the site, the more effective the chatbot is in providing information to users and achieving their goals.
Satisfaction with response
Satisfaction with customer service using chatbots. Set up a simple questionnaire and measure it after the chatbot answers.
When measuring, asking questions that can be answered with “yes” or “no” makes it easier to visualize satisfaction. An example survey question might be, “Are you satisfied with the answers?”
Number of inquiries for manned support
If the number of manned inquiries such as telephone support is decreasing, it is considered that the user was able to solve the problem only with the chatbot. Therefore, after introducing the chatbot, be sure to measure the number of manned inquiries.
At the same time, if you measure changes in the working hours of people in charge such as operators, you can check whether or not expenses are being reduced. If a wide range of user problems can be solved only by chatbots, problems can be solved efficiently, and the work burden on the entire person in charge should tend to be reduced.
Procedures for measuring chatbot effectiveness using evaluation metrics
There are five steps to measuring the effectiveness of chatbots using evaluation metrics. By following each step carefully, you will be able to maximize the effectiveness of your chatbot.
Step 1. Clarify the purpose of chatbot operation
When introducing a chatbot, you have to think about the purpose of “for what purpose?” Therefore, first clarify the purpose of introducing a chatbot and set the goal of verifying its effectiveness.
Goals include “reduction of call volume to the call center” and “improvement of operational efficiency of operators”. Setting clear goals will help you understand what features you should implement in your chatbot.
Step2.Determine the evaluation indicators necessary for effect measurement
Next, let’s set an appropriate evaluation index according to the purpose. Since the necessary indicators differ depending on the type and purpose of the chatbot, it is important to clarify the purpose in Step 1.
Examples of evaluation metrics include “response rate”, “correct answer rate”, and “resolution rate”. It is important to decide the indicators that should produce results against the goals based on the evidence.
Step3. Set target values for evaluation indicators
Once you have determined the indicators that should produce results for your goals, set numerical targets for each indicator. After setting the final target value, set KPI to achieve it. For example, if the goal is to “reduce the number of manned responses for all operators by 10%”, KPIs such as “number of activations”, “response rate”, and “resolution rate” will be set.
It will be easier to create a chatbot that meets user needs by clarifying the areas where improvements are lacking while looking at the KPI figures.
Step4.Actual data collection and measurement
After setting the KPIs, let’s actually collect the data of each indicator and measure it. Once you start collecting data, you will know what improvements to make.
However, unless it’s a big problem, let’s try to improve it little by little. Especially for chatbots, it is necessary to investigate user movements, so if you make too many improvements at once, you may not be able to understand the underlying problem. By making only one change per improvement, it becomes easier to visualize the problem areas of the chatbot.
Step5. Compare the data before and after introducing the chatbot
It is important to compare post-chatbot and pre-implementation data. Chatbots are introduced with the expectation of introduction effects and benefits. Therefore, you need to know how far you have reached your goals. If the pre-implementation numbers are good, there may be some problem with the chatbot service, so be careful.
Step6. Consider measures for improvement
Data is not only for viewing, but also an element for considering improvement measures. It is important to make a hypothesis about where the problem is in the chatbot while looking at the data after introduction, and search for improvement methods. Also, by repeating the measures many times instead of just once, the accuracy of the response will improve.
Evaluation metrics are important when introducing a chatbot
When purchasing a chatbot, be sure to determine the evaluation metrics. However, it is important to calculate backwards from the final goal rather than suddenly deciding the purpose. In addition, evaluation indicators include various numerical values such as “response rate” and “resolution rate”. In order to visualize what went wrong, it is important not to improve everything at once, but to improve little by little while observing user behavior.
