Mental Models for Intelligent Assistants - NN/G

Introducing NN/G Membership | Access all self-paced courses for $249 per year | Join by Jan. 26, 2027 |
Find the training option that fits your learning style.
February 3, 2019 2019-02-03
In previous research, we found that even frequent users of intelligent assistants (IAs) such as Siri, Alexa, or Google Assistant use them for just a small number of simple tasks: getting weather or news information, playing music, setting alarms, timers, or reminders, and answering trivia questions. This limited usage reflects the poor usability of these assistants, which are still far from addressing real user needs.
One of the dangers that we warned against was that people will get used to these poorly performing assistants and, even when the assistants will become better, users may not discover their improved capabilities.
In this article, we report research that examines users’ mental models and expectations for intelligent assistants. In particular, we looked at the mental-model differences between frequent users of IAs and new users, aiming to understand to what extent the views about IAs’ strengths and limitations shape the usage of these systems.
We carried out a 2-week diary study with two groups of intelligent-assistant users:
Both user groups had to log 16 unique interactions with their assistant during the two weeks of the diary, with a minimum of 8 interactions per week. Participants could not record the same type of activity twice — for example, they could not use the assistant to navigate to a destination twice (even if the destination was different for the two instances).
For each diary entry, users had to respond to a few questions regarding their goal, including whether the assistant helped them complete their goal and whether they were satisfied with the assistant’s help. Participants also performed a mental-model elicitation task three times during the study: before they started the diary, after the first week, and at the end. The mental-model elicitation consisted of a series of questions that the participants had to answer about their assistant.
Our goal was to understand how people evolve their mental models and expectations of the assistants as they expanded the range of activities that they performed with them.
The requirement to log only new types of activities was challenging for our participants. While the first week was generally successful, in the second week people started to comment that it was hard to find new things to do with the assistant. This finding confirmed our previous research, which had shown that users engage in a fairly limited set of simple activities with their assistants (despite the numerous and much advertised skills that some of these assistants have — in theory, not in actual use).
Many of our participants started to actively explore the tasks their assistants could do — sometimes by checking external sources like the web or newsletters about their assistants’ capabilities.
For each activity that participants logged, they had to say whether the IA completed the activity successfully and whether they were satisfied with their assistant’s performance in that task. The average IA completion rate (as perceived by the users) for the 2 weeks was 58%. The average completion rate in the first week was 60% and in the second week was 58%.
An analysis of the various IAs found no statistical difference for the completion in week 1 versus the completion in week 2, with one exception: that of Google Home devices. Google Assistant’s performance on Google Home devices deteriorated significantly from 64% in week 1 to 44% in week 2 (p < 0.005).
The satisfaction data mirrored the completion data. The satisfaction measures we report are the percentage of activities for which people said they were satisfied with the IA’s performance. In week 1 satisfaction averaged 76%; in week 2 it was 72%. The only statistically significant difference in the satisfaction between the two weeks was for Google Home devices — in week 1, the average percentage of satisfying interactions was 86; in week 2, the number dropped to 63% (p <0.05).
We tried to understand why Google Home owners had lower completion and satisfaction scores in the second week. (Usability research virtually always records higher success and satisfaction as users gain more experience, so this outcome was a surprise.) In their general comments about their understanding of the system, out of the 6 Google Home participants, 4 expressed disappointment and said they found out about some limitation of the assistant. Here are some of their comments:
Out of the other two participants, one was overall positive, and another one was pleased with the system, and mentioned just one negative fact (“Google still doesn't totally understand kids. Right. Sometimes it's easier for mommy to talk to Google than you.”)
Based on these comments (as well as on the data that we report in the following sections of this article), we surmise that declining success rate and satisfaction might have been caused by the new tasks that people had to try during the second week and that weren’t up to their expectations. It’s somewhat surprising however that the second week did not affect the frequent users of the other IAs, as well. To better understand why that was the case, we looked at the types of activities that users did. To increase the number of new activities, most IA users resorted to fact-finding tasks, which were the easiest way to expand their assistants’ realm of use. Not Google Home users — compared with the other IA users, these participants had just a minor increase in fact-finding activities. (In week 2, Google Home users had significantly fewer fact-finding activities than Siri users, p <0.005, Google Assistant users on iPhone, p< 0.0001, and marginally fewer than Alexa users, p =0.07.)
Because fact-finding activities are relatively successful (provided that users formulate their questions adequately), they did not cause too much disappo