Behind the Curtain: Wizard of Oz Method for Design Studies
There are a large variety of research methods used to gather data on user experience and interaction with a product or design. Some take place in lab settings while others seek out real world environments for contextual information.
One in particular that gives users the experience of interacting on their own is called the Wizard of Oz method.
Wizard of Oz Method?
Unlike other research methods, the name “Wizard of Oz” sounds unusual. However, the actual format of this technique offers a unique and insightful way for learning about user experience and interaction.
In short, much like how the Wizard of Oz in the book/movie was revealed to be a man operating the machine behind a curtain, users interact with what they believe to be an autonomous interface or design that is actually controlled by a researcher.
Design Studies
An interesting example of the Wizard of Oz method is the research study Vero: An accessible method for studying human-AI teamwork by Schecter, Aaron et al. Researchers wanted to see how people interacted with an AI team member in a Zoom video conference and how their AI model could be used for future Wizard of Oz studies.
Vero model in Zoom
The AI teammate, named “Vero,” is a simple 3D model that occupies a slot in the video conference to interact with the human users.
The researchers explain that “The Vero Method relies on three components: A video conferencing platform that allows the use of virtual animated backgrounds (e.g. Zoom), a set of animations for each of the non-verbal behaviors the agent can perform, and an introduction video that establishes the belief in participants that artificial agents that act as teammates and can converse in ways indistinguishable from people are a technical possibility” (Schecter, Aaron et al.)
As they stated, Vero has a set of basic animations that the “wizard” can use to indicate actions such as listening, speaking, or waving. They do so by switching their Zoom background to the appropriate animation.
Vero’s animations
In a case study, the researchers placed “wizards” as Vero into groups of 2 to 3 people where they had to complete a few simple team collaboration tasks.
Results showed that 91.67% of the participants believed Vero was an intelligent AI and only 8.33% thought it had been a human.
Common themes in responses to why they believed Vero was AI
Users viewed Vero as similar to existing chatbot systems, increasing believability. “On the other hand, this familiarity seems to limit participants’ views of Vero’s capabilities. In other words, participants assumed that Vero – like any other machine or bot – is simply an algorithm responding to prompts with fixed phrases. As a result, some participants did not treat Vero as a normal teammate” (Schecter, Aaron et al.) Despite this, changes to the wizards’ scripts did impact how the teams functioned, meaning AI team members like Vero can have useful effects as part of a group.
The researchers hope that their Wizard of Oz model can be useful in future studies about interactions between humans and AI in group work.
Sources:
Schecter, Aaron et al. “Vero: An accessible method for studying human-AI teamwork.” Comput. Hum. Behav. 141 (2022): 107606.
