Remote User Testing for Learning Prototypes

The two projects I’m working on currently have a global audience, many of whom are located in areas with less-than-ideal Internet connections. Earlier this year, I tested several sets of prototypes with about 30 of these learners total.

Remote prototype testing became much easier for me with time, so I wanted to collect some of the technologies and practices that helped me, in case they are helpful to someone else too.

Testing the Interface Design vs. Testing the Learning Design

First, I wanted to share just a few thoughts why I test prototypes with learners.

As part of my typical design process, I create click-able prototypes of the planned interactions and test them with members of the target audience. While this is partially user interface testing, mostly what I’m testing is the thought process of the learners as they approach the activities: I want to know whether they are pondering what I hope they are pondering. Seeing these early reactions helps me adjust both the interfaces and the content to better meet the goals of the project.

During this testing, I have learners complete the prototypes while I watch and take notes. I encourage them to think out loud about why they are making certain decisions. While I avoid offering guidance, I do periodically stop learners to ask what they think is the main point of an activity, or why they made one choice over another.

Until this year, most – though not all – of the prototype testing I’d conducted had been in person, where it was easier to observe actions and reactions. During these rounds of remote testing, I found that I needed to ask more questions about their experience than for in person testing.

Scheduling Sessions

Scheduling for many different time zones was a challenge, especially since I was in California (USA) while most of the testers were in Europe, Africa, and Asia. I kept my schedule open for early morning and late evening appointments and used Doodle to give a range of possible times that testers could select from. Doodle automatically adjusts to the time zone of each person, meaning that I spent less time converting times (and had less opportunities to make time-zone mistakes.)

Preparing for Sessions

English wasn’t the first language of most of the testers and set up was often the most confusing part, so I began sending instructions in advance with links to the prototypes and instructions on how to start Skype screen sharing.

Before we started, I would also open a Word template for note taking that listed all of the prototypes I planned to test as well as other more general questions. While I typically took notes about sessions on my computer, I would also keep paper nearby in case we had to switch to using my computer to access the prototypes (more about this below.)

Connecting to Testers

For the actual sessions, I generally tried to use Skype audio with screen sharing since it’s free and fairly widely used. I’d have learners load the prototypes and start screen sharing to let me watch and listen as they completed the activities.

This worked for most but not all of the sessions:

  • For two testers, Skype stopped working altogether (possibly, throughout the whole country.) For two other testers, Skype audio mysteriously failed (video and chat was fine.)
  • Two people had versions of Skype that didn’t seem to offer screen sharing.
  • For one person, the prototypes wouldn’t load at all.

To help with some of these challenges, I also signed up for a account. audio options were too limited/confusing for our needs, but the screen sharing software allowed for remote control of a shared screen – so, in addition to watching learners completing the prototypes on their computers, I could load the prototypes on my computer and give the learner control as needed.

Between the two projects, I had over 30 sessions. By the end, technical problems stopped fazing me. If we had a problem, instead of troubleshooting, I just switched technologies. So, if Skype audio failed, I’d ask for a phone number. If they didn’t have a phone number, we’d use text to chat. If a prototype wouldn’t load on their machines, I’d give them remote access to my machine. I was asking people to take time out of their day, and spending an hour troubleshooting why something wasn’t working was a waste of everyone’s time.

New eLearning Global Giveback Course: Isoniazid Preventive Therapy for the Prevention of Tuberculosis in People Living with HIV

My son, Theo, was born two weeks ago, so I won’t be at the Learning Solutions Conference this week. However, a course I developed with FHI 360 is going to be demoed there on Thursday evening as part of the third eLearning Global Giveback.

This course is designed to give health care workers hands-on practice identifying when and how an Isoniazid Preventive Therapy (IPT) program can be used to help prevent tuberculosis in people living with HIV.

The content for the course is two documents: a set of WHO guidelines and a brief created by FHI 360. I purposefully didn’t try to recreate this material in the course itself, because I felt like the searchable, skimable PDFs were already far better as lasting resources than an online course would be. Instead, I included the documents as downloadable resources and focused the course itself on application of knowledge.

The course is a scenario-based practice space where the user can apply the information from the downloaded documents in similar-to-real-life activities. At a first medical center, the user begins by screening a series of patients to determine whether or not they are good candidates for IPT and another series of patients who are already undergoing IPT to determine if they are exhibiting side effects of the therapy.

Once the user has completed the screening exercise, they address common misconceptions about IPT with some of the staff at this medical center.

The second location is a smaller, more rural clinic. Here the user also screens patients to determine if they are good candidates for IPT, but without access to a screening test for tuberculosis infection.

Throughout the design and development process, the team at FHI 360 provided excellent support and reviews, and I’m really excited about how the course turned out.  If you aren’t able to stop by the Global Giveback reception to demo the course, you can take a peek at the course in my online portfolio. You can also learn more about the eLearning Global Giveback by clicking here!

Interface Inspiration

I’m committed to online learning that is focused on activity, typically of the real-world, application-focused variety.  However, imagining how to translate real-world situations into meaningful learning interfaces for a computer screen can sometimes be daunting. Even in the 21st century, significant portions of our lives still don’t revolve around pointing and clicking.

One strategy I use is keeping an “interface inspiration” library – essentially, a folder on my computer where I keep screenshots of learning interfaces that I think are particularly nifty.  When I get stuck while trying to translate activities into interfaces, I can browse through my collection of screenshots for ideas.

I mentioned this strategy a few weeks ago while presenting as part of an instructional design panel at DevLearn, and the question of WHERE to find these interfaces came up during the Q&A portion of the session.  In a sort of belated cyber answer (yes, I answered in person too), here is a preliminary list of places where I’ve found inspiration for learning interfaces.

The sites below are a mix of learning venders and educational game sites, but all include multiple clickable demos that I found useful. Most of the learning games sites target school-age children, but the interface ideas are absolutely still relevant for learning aimed at adults too.

Allen Interactions

I think that Allen Interactions creates some of the best performance-driven, application-focused eLearning available, and they have a number of courses with amazing interfaces on their web site. You do need to sign up for an account to view their case studies and demos, but it’s quick and free.

Filament Games

Filament Games is my current educational games company crush. The link above goes straight to their large library of games that you can play – the iCivics games are probably my favorites.

Enspire Learning

Enspire Learning is another learning-focused company with some great demonstrations of intuitive yet creative interfaces on their web site. Again, handing over your contact information is required to view the demos.

Doorways-to-Dreams Foundation – Financial Entertainment

I have mixed feelings about casual games for learning, but Doorways-to-Dreams has a good series of financial entertainment games created by several different venders. My favorite for interface ideas is Celebrity Calamity.


EdHeads in a nonprofit organization that designs and develops really neat educational online games focused on science, math, and critical thinking.

Nobel Prize – Educational Games

This section of the official site of the Nobel Prize offers a series of educational games based on Nobel-prize achievements over the years.

In additional to these initial links, I’ll add future updates to the newly created Interface Inspiration page found through the blog menu above.

DevLearn 2011

Both a plethora of work-type projects and some significant (but exciting) life changes conspired to keep me busy and poof, now it’s almost November, summer is officially over, and I’ve managed to not post here for months.

Oops. I’ll try to do better.

Right now, I am finishing up a couple of nifty projects that I plan to post about soon (I promise!)  In the meantime, I just thought I’d quickly post to say that I will be at DevLearn 2011 next week.

During the Wednesday 2:45 pm concurrent sessions, I’m presenting as part of a panel: One Learning Challenge – Three Designers Put Their Skills to the Test. The session should be a lot of fun, and I’m looking forward to seeing the designs of the other two panelists, Judy Unrein and Carol Ann Amico. On Thursday evening from 4 to 7 pm, I’m sharing my serious game Flower Stand as part of DemoFest.

If you’ll be at DevLearn too, let me know, and I’d love to catch up in person!

Beyond Points, Badges, and Scoreboards

In the past, when I’ve told people that I’m interested in serious games for learning, I’d often need to clarify, “not like online Jeopardy,” because quiz show games had become the standard in so many people’s minds.  Now, that gamefication is super-trendy, the discussions often revolve around whether or not to pepper online courses with points, scoreboards, and badges.

I generally lean towards not.

It’s not that I think the gamefication of learning is necessarily bad. I just think that most of the time, we can do so much better.

Instead of essentially trying to trick the learner to stay engaged through extrinsic rewards like points or badges, I’d rather spend my energy ensuring that the learning activities are themselves engaging and useful. I can still see elements of gamification as periodically useful, say to help motivate the user through dry, introductory material that is necessary to master before they moving on to more complex material. However, I’m wary of gamification when it becomes an overall strategy for learning design.

Along those lines, I was excited when the GDC Vault posted Jane McGonigal’s GDC 2011 presentation, We Don’t Need No Stinkin’ Badges: How to Re-invent Reality Without Gamification. The video is an hour long, but it’s absolutely worth the time.

As part of the presentation, McGonigal distinguishes between:

  • Gamification – making something LOOK like a game by giving you points, levels, badges, and high scores to convince you to do things that you might otherwise not want to do (extrinsic rewards)
  • Gameful Design – making something FEEL like a game by generating positive emotions, relationships, meaning, and accomplishments (intrinsic rewards)

McGonigal argues that gameful design, rather than gamification, is more likely to result in games with a positive impact on the user’s lives and the world.

Similarly, I think that learning that follows focuses on these same intrinsic rewards is more likely to have a real-world, long-term impact.