A simple task to assess the human-likeness of chatbots
New chatbots are introduced every day to help people in a variety of settings. Even though scientists have been studying chatbots since the 1960s, it is still unknown which social features are needed to make a chatbot human-like. This study presented at the Workshop on Chatbot Research and Design introduces a new simple task with which we can successfully measure the human-likeness of chatbots.
Take aways
- With a simple task asking people to describe objects to a text-based chatbot, we can easily assess whether people perceive a chatbot as human-like or not.
- It works like this: When people take the chatbot’s perspective into account into their descriptions they ascribe some human-likeness to the chatbot—even without realizing they are doing so.
- Due to the simplicity and the efficiency of this task, it can serve to evaluate the communication of chatbots and compare it with human communication.
Study information
Who?
39 students from a Dutch university (mean age: 19 years; 82% women)
Where?
The Netherlands
How?
The participants interacted with a text-based chatbot. First, they had to ask the chatbot for movie information to make sure they interacted with it for a certain period of time. Then, they performed a description task with the chatbot in which they had to label objects. Researchers looked whether participants took the chatbot’s perspective into account in their description. As a control, participants performed the description task alone, without the chatbot. The results were compared with an earlier study where the participants needed to describe the objects for a real, human, person.
Facts and findings
- The participants labeled objects differently when doing the task with the chatbot than when they did the task alone. This indicates that they took the chatbot’s perspective into account. Participants believed the chatbots would only understand them if they described the objects in a certain way.
- Participants in this study took the chatbot’s perspective into account less than the participants in the earlier study did, taking another human’s perspective.