What Session Length Can Tell You About Your Chatbot Performance

If you’re looking to improve your chatbot, it’s first necessary to identify what it does well and where it’s failing to live up to expectations. We do this by using specific metrics to evaluate the chatbot’s performance. Many of these metrics will be familiar to those who have experience of telephony-based customer service. Others are unique to chatbots.

One of the most important metrics employed by chatbot developers is average session length. This one goes by a lot of different names; you may also commonly hear it referred to as Average Handling Time (AHT). Here, we take a look at what session length can tell you about your chatbot’s performance and how it can be used to identify flaws and improve customer service provision.

Defining chatbot session duration

Before we get started on the details, it’s helpful to establish a comprehensive definition of session length. At its most basic, session length is the amount of time that passes between the moment a user first interacts with a chatbot and the moment they navigate away from the chatbot or terminate the interaction.

Average session length is just as simple. If you take all of your chatbot conversations over a period of time (it could be a day, week, month or year – that’s up to you), add together the length of all interactions and then divide by the total number of interactions, you’ll get the average session length.

What Session Length Can Tell You About Your Chatbot Performance

For instance, imagine you have three conversations in a day. The first lasts 2 mins 35 secs, the second 5 mins and 5 secs and the third just 25 secs. Your calculation for average session length would be as follows.

215 secs + 305 secs + 50 secs = 9 mins 30 secs
9 mins 30 secs / 3 interactions = 3 mins 10 secs
Average Session Length = 3 mins 10 secs

What constitutes a good session length?

Now you have a means of working out your average session length, you’ll probably want to know what constitutes a good session length.

Unfortunately, it’s just not that easy. Generalisations can be made about session length that will, by and large, stand true. For instance, when analysing session length there’s generally an optimal ‘window’ that represents a ‘good’ session length.

If your chatbot is recording session lengths that are short of this window it may indicate that your bot is frustrating users and causing them to navigate away before their enquiry is resolved. On the other hand, long sessions may suggest that your bot is taking too much time to complete tasks. This will frustrate users too, making them feel as though the bot is wasting their time.

Session length – not as simple as it seems

However, there are no established parameters for this optimal window. In other words, the ideal session length varies from bot to bot and will largely depend on what the chatbot’s primary function is and the context of the conversation.

Some chatbots are designed to complete enquiries in as short a period of time as possible. A good example of this would be a chatbot that helps users book train tickets. In this instance, the bot has a single, basic function and requires relatively little information to complete the task. The aim here is for the user to successfully complete the transaction as quickly as possible.

Other chatbots require more time. For example, a chatbot that helps users with shopping by recommending items will want to keep that user engaged for a prolonged period.

What Session Length Can Tell You About Your Chatbot Performance

Setting parameters

Ultimately, you’ll have to set the parameters for your chatbots optimal window. This can be achieved by looking at all of your interactions over a set period and marking them as either successfully resolved or incomplete. If your sample size is large enough, you should start recognising a general pattern. Successful interactions will sit within the window, while failures will mark the limits of your window.

It’s important to note that you’ll still have incomplete sessions that sit within your window and successful interactions that fall outside of it. These are outliers. What you’re looking for is a general pattern that makes it easier to identify when your chatbot is not performing as it should.

As well as marking the chatbot interactions as successful or unsuccessful yourself, it’s also a good idea to get customer feedback. Using customer satisfaction scores, you can assess whether your analysis of an interaction matches up with the customer’s view. After all, a chatbot can resolve a customer’s enquiry without that customer feeling as though the bot provided satisfactory service.

Using session length to improve services

The most useful ways to leverage your session length metric as a way to improve service are to:

  1. Use session length to identify failed interactions – any interactions that fall outside the optimal window can be checked to see what went wrong
  2. Use session length as an alarm system – if you see an increase in session lengths outside of your optimal window, you know that users are interacting with your bot in a different way or that there’s an issue that needs to be resolved
What Session Length Can Tell You About Your Chatbot Performance

Interactions that fall outside the optimal window

By looking at those interactions that fall outside of the optimal session length window, you can identify where they went wrong and make changes to improve the service. In such cases, you want to ask:

  1. Should these interactions be classified as failures?
  2. Where does the bot fail and why?
  3. How could the flaw be resolved and the customer service improved?

What Next?

Taken on its own, session length is a powerful warning system and a means of identifying problematic interactions. However, it really comes into its own when analysed in conjunction with other chatbot metrics. By pairing it with other metrics, it provides you with a more detailed picture of how your chatbot is performing, allowing you to make adjustments that maximise benefits to your organisation.

Have a question or want further information on improving customer service using chatbots? Our expert team have been providing customer contact solutions for over 25 years. Call us on 01344 595800 or drop us a line.