Remote User Testing for Learning Prototypes

The two projects I’m working on currently have a global audience, many of whom are located in areas with less-than-ideal Internet connections. Earlier this year, I tested several sets of prototypes with about 30 of these learners total.

Remote prototype testing became much easier for me with time, so I wanted to collect some of the technologies and practices that helped me, in case they are helpful to someone else too.

Testing the Interface Design vs. Testing the Learning Design

First, I wanted to share just a few thoughts why I test prototypes with learners.

As part of my typical design process, I create click-able prototypes of the planned interactions and test them with members of the target audience. While this is partially user interface testing, mostly what I’m testing is the thought process of the learners as they approach the activities: I want to know whether they are pondering what I hope they are pondering. Seeing these early reactions helps me adjust both the interfaces and the content to better meet the goals of the project.

During this testing, I have learners complete the prototypes while I watch and take notes. I encourage them to think out loud about why they are making certain decisions. While I avoid offering guidance, I do periodically stop learners to ask what they think is the main point of an activity, or why they made one choice over another.

Until this year, most – though not all – of the prototype testing I’d conducted had been in person, where it was easier to observe actions and reactions. During these rounds of remote testing, I found that I needed to ask more questions about their experience than for in person testing.

Scheduling Sessions

Scheduling for many different time zones was a challenge, especially since I was in California (USA) while most of the testers were in Europe, Africa, and Asia. I kept my schedule open for early morning and late evening appointments and used Doodle to give a range of possible times that testers could select from. Doodle automatically adjusts to the time zone of each person, meaning that I spent less time converting times (and had less opportunities to make time-zone mistakes.)

Preparing for Sessions

English wasn’t the first language of most of the testers and set up was often the most confusing part, so I began sending instructions in advance with links to the prototypes and instructions on how to start Skype screen sharing.

Before we started, I would also open a Word template for note taking that listed all of the prototypes I planned to test as well as other more general questions. While I typically took notes about sessions on my computer, I would also keep paper nearby in case we had to switch to using my computer to access the prototypes (more about this below.)

Connecting to Testers

For the actual sessions, I generally tried to use Skype audio with screen sharing since it’s free and fairly widely used. I’d have learners load the prototypes and start screen sharing to let me watch and listen as they completed the activities.

This worked for most but not all of the sessions:

  • For two testers, Skype stopped working altogether (possibly, throughout the whole country.) For two other testers, Skype audio mysteriously failed (video and chat was fine.)
  • Two people had versions of Skype that didn’t seem to offer screen sharing.
  • For one person, the prototypes wouldn’t load at all.

To help with some of these challenges, I also signed up for a join.me account. Join.me audio options were too limited/confusing for our needs, but the screen sharing software allowed for remote control of a shared screen – so, in addition to watching learners completing the prototypes on their computers, I could load the prototypes on my computer and give the learner control as needed.

Between the two projects, I had over 30 sessions. By the end, technical problems stopped fazing me. If we had a problem, instead of troubleshooting, I just switched technologies. So, if Skype audio failed, I’d ask for a phone number. If they didn’t have a phone number, we’d use text to chat. If a prototype wouldn’t load on their machines, I’d give them remote access to my machine. I was asking people to take time out of their day, and spending an hour troubleshooting why something wasn’t working was a waste of everyone’s time.