Remote User Testing for Learning Prototypes

The two projects I’m working on currently have a global audience, many of whom are located in areas with less-than-ideal Internet connections. Earlier this year, I tested several sets of prototypes with about 30 of these learners total.

Remote prototype testing became much easier for me with time, so I wanted to collect some of the technologies and practices that helped me, in case they are helpful to someone else too.

Testing the Interface Design vs. Testing the Learning Design

First, I wanted to share just a few thoughts why I test prototypes with learners.

As part of my typical design process, I create click-able prototypes of the planned interactions and test them with members of the target audience. While this is partially user interface testing, mostly what I’m testing is the thought process of the learners as they approach the activities: I want to know whether they are pondering what I hope they are pondering. Seeing these early reactions helps me adjust both the interfaces and the content to better meet the goals of the project.

During this testing, I have learners complete the prototypes while I watch and take notes. I encourage them to think out loud about why they are making certain decisions. While I avoid offering guidance, I do periodically stop learners to ask what they think is the main point of an activity, or why they made one choice over another.

Until this year, most – though not all – of the prototype testing I’d conducted had been in person, where it was easier to observe actions and reactions. During these rounds of remote testing, I found that I needed to ask more questions about their experience than for in person testing.

Scheduling Sessions

Scheduling for many different time zones was a challenge, especially since I was in California (USA) while most of the testers were in Europe, Africa, and Asia. I kept my schedule open for early morning and late evening appointments and used Doodle to give a range of possible times that testers could select from. Doodle automatically adjusts to the time zone of each person, meaning that I spent less time converting times (and had less opportunities to make time-zone mistakes.)

Preparing for Sessions

English wasn’t the first language of most of the testers and set up was often the most confusing part, so I began sending instructions in advance with links to the prototypes and instructions on how to start Skype screen sharing.

Before we started, I would also open a Word template for note taking that listed all of the prototypes I planned to test as well as other more general questions. While I typically took notes about sessions on my computer, I would also keep paper nearby in case we had to switch to using my computer to access the prototypes (more about this below.)

Connecting to Testers

For the actual sessions, I generally tried to use Skype audio with screen sharing since it’s free and fairly widely used. I’d have learners load the prototypes and start screen sharing to let me watch and listen as they completed the activities.

This worked for most but not all of the sessions:

  • For two testers, Skype stopped working altogether (possibly, throughout the whole country.) For two other testers, Skype audio mysteriously failed (video and chat was fine.)
  • Two people had versions of Skype that didn’t seem to offer screen sharing.
  • For one person, the prototypes wouldn’t load at all.

To help with some of these challenges, I also signed up for a account. audio options were too limited/confusing for our needs, but the screen sharing software allowed for remote control of a shared screen – so, in addition to watching learners completing the prototypes on their computers, I could load the prototypes on my computer and give the learner control as needed.

Between the two projects, I had over 30 sessions. By the end, technical problems stopped fazing me. If we had a problem, instead of troubleshooting, I just switched technologies. So, if Skype audio failed, I’d ask for a phone number. If they didn’t have a phone number, we’d use text to chat. If a prototype wouldn’t load on their machines, I’d give them remote access to my machine. I was asking people to take time out of their day, and spending an hour troubleshooting why something wasn’t working was a waste of everyone’s time.

Other Ways to Inspire Change

What interests me most about learning is its potential to inspire change: ideally a measurable change in behavior, but barring that, at least a shift in attitude or understanding.

Very often, we decide we want something to change, we decide to create a course, lesson, document, or some other package of information that learners will absorb and, voila, change!

What if, though, in many cases it’s not actually a lack of information that’s preventing learners from doing things differently?

A couple of weeks ago, I was pondering how to create educational games that allow learners to practice avoiding common advertising tricks. I had hit a wall after play testing several game prototypes. Simultaneously, I was also lamenting the existence of holiday gifts, and how much pressure there is to buy meaningful gifts for an ever-widening circle of receivers within a commercial sea of expensive candles and scented lotions.

That was the moment of inspiration for Gift Instead, my latest project.

What if, instead of trying to teach about advertising, we just gave people tools to opt out of the whole purchase-more-oriented system – well, at least for giving and receiving holiday gifts this year?

Gift Instead is a web site that lets users create, customize, and share purchase-free gift lists. Users can explore prewritten ideas for purchase-free gifts, but they can also create their own.


My awesome husband Brian did most of the heavy lifting with piecing together a framework for the site and was my brainstorming partner in making this project come alive.

I’m really excited about the site we created. It’s still officially in beta mode, but it’s a very functional beta: please feel free to try your hand at creating and sharing a wish list and do send me any feedback about your experience.

I’ll probably revisit the idea of educational games about advertising schemes in the future. However, I really enjoyed working on another potential path into that same sort of change – less needless stuff purchased – by creating a tool rather than a packaged educational experience.


An Unfinished Election Game (Or, you can’t win everything all of the time)

I started creating a serious game during maternity leave that is likely going to remain unfinished now that I’m back to work full-time. However, I did want to at least capture some of my thoughts about the project here.

Essentially, I was just looking for a fun project to help maintain my sense of sanity while taking care of my newborn son.

Newborns. Cute, but not always good for my sanity.

Thanks to an episode of This American Life, I stumbled upon a blog post by the Sunlight Foundation that explores the fundraising potential of various house committees.  The This American Life episode further explores just how much time, effort, and wooing of lobbyists actually goes into getting re-elected to Congress. I’m a huge fan of numbers and relationships, so I decided to dive right in, either by creating a serious game or an interactive graphic.

Numbers. Not as cute, but much better for my sanity.

Because I entirely overestimated the amount of time I’d have to work on the project, I went the serious game route. (Babies take up lots of time! Who knew?)

First, you select a candidate. Eventually, I planned to ambiguate (made-up word for the post) the text items that appear on each platform. For early development, I picked categories that translated pretty directly into membership for certain committees.

After selecting a candidate, you walk through some introductory material about your background as a junior congressperson who just barely won because your opponent was exposed in a last minute scandal… and information about just how much fundraising you’ll have to do in order to get re-elected as an underdog candidate.

You get some advice!

Then you get to select your committee preferences…

… and then see how the committees you were actually selected for impact your fundraising potential.

Your preferences and experience are taken into account.  Of course, as a not-very-important junior congressperson, there isn’t a chance that you’ll end up on the Ways and Means committee. Sorry.

Once you’ve been assigned to your committees, the real fundraising work begins!  You select the number of hours you want to dedicate to various activities.  Depending on your choices, your popularity with your constituents, lobbyists, and family will rise and fall.

You also have a number of special activities, represented by click-able items on the desk – emailing your constituents for donations, calling a lobbyist and asking them to hold a fundraiser for you, and mortgaging your house.

However, it’s often not as straightforward as just asking. No one is going to hold a fundraiser for you if you haven’t proven your usefulness.  If you don’t have strong support from your constituent base, emailing them won’t result in donations.

Decide to mortgage your house and spend all of your free time meeting with lobbyists?  Well, your reception at home next month might be a bit chilly.

All of these actions, plus your committee membership and unimplemented special events (say, voting on a particular bill that affects a potential donor) would affect your ability to raise money toward your next election.

That’s the unfinished game. Here are my thoughts on how it might have all pulled together.

I love the philosophy that the strategy for winning a serious game should be the message you’d like the learner to remember. I especially love the idea of this being a semi-hidden message that the user only discovers through trial and error during the gameplay itself. For this election game, I wanted the message not just to be about the committees and numbers (as it appears on the surface), but also about sacrifice and life choices (this is the secret message part… shh!).

In Ralph Koster’s A Theory of Fun, he talks about how we need more games around today’s challenges versus knights and swords. Though I’m not opposed to medieval warfare, I also love the idea of games that address the same sorts of conflicts I face in my modern life.

Generally, when I play games, I want to maximize my winning. My Sims reach the top of their careers, have great relationships with their families, throw amazing parties, and manage to have the biggest house on the block. However, in real life, winning at everything isn’t always possible.  So, in the finished version of the election game that exists only in my head, the user would be encouraged to try to balance work and family life…

… but the only actual way to win the election as an underdog candidate would be to not achieve this work life balance. You can retain your congressional seat, but only at the cost of a divorce. This isn’t meant to be a statement about the impossibility of having a happy family and a successful career (gulp!), but I did want to play with the idea that you can’t win at everything all of the time (unless you are one of my Sims).

Though the chances of me having time to finish this game in the near future are less than your character’s chances of being appointed to a lucrative congressional committee, you can still go here and click around my unfinished prototype, should you so desire!  Be warned though: winning is a feature that has not yet been implemented.


New eLearning Global Giveback Course: Isoniazid Preventive Therapy for the Prevention of Tuberculosis in People Living with HIV

My son, Theo, was born two weeks ago, so I won’t be at the Learning Solutions Conference this week. However, a course I developed with FHI 360 is going to be demoed there on Thursday evening as part of the third eLearning Global Giveback.

This course is designed to give health care workers hands-on practice identifying when and how an Isoniazid Preventive Therapy (IPT) program can be used to help prevent tuberculosis in people living with HIV.

The content for the course is two documents: a set of WHO guidelines and a brief created by FHI 360. I purposefully didn’t try to recreate this material in the course itself, because I felt like the searchable, skimable PDFs were already far better as lasting resources than an online course would be. Instead, I included the documents as downloadable resources and focused the course itself on application of knowledge.

The course is a scenario-based practice space where the user can apply the information from the downloaded documents in similar-to-real-life activities. At a first medical center, the user begins by screening a series of patients to determine whether or not they are good candidates for IPT and another series of patients who are already undergoing IPT to determine if they are exhibiting side effects of the therapy.

Once the user has completed the screening exercise, they address common misconceptions about IPT with some of the staff at this medical center.

The second location is a smaller, more rural clinic. Here the user also screens patients to determine if they are good candidates for IPT, but without access to a screening test for tuberculosis infection.

Throughout the design and development process, the team at FHI 360 provided excellent support and reviews, and I’m really excited about how the course turned out.  If you aren’t able to stop by the Global Giveback reception to demo the course, you can take a peek at the course in my online portfolio. You can also learn more about the eLearning Global Giveback by clicking here!

What Will Be Left Then?

Even though I generally play more with creating learning activities than representing information, I’m fascinated by data visualization.

A couple of weeks ago, I created an infographic as part of Information is Beautiful’s first visualization challenge.

Information is Beautiful provided a spreadsheet of data about when the reserves of various non-renewable resources will be depleted. I carried the data in my handbag and periodically pulled it out to peer at it while waiting at doctor’s offices, restaurants, and the like.  However, because of various commitments and some possible procrastination on my part, I ended up with only to one day to actually design and develop the visualization in Flash. I decided to go for it anyway.

Here’s a screenshot from the visualization I created:

If you click on the image (or here), you can view the interactive version in my portfolio. As the viewer clicks the timeline to view different stages of a person’s life, the resources at the bottom of the screen gray out as the reserves for each are projected to be extinguished.

I wanted to focus on the emotional aspects of resource depletion and decided to frame the depletion of resources around the life of a child born in 2011.  Perhaps in part because my husband and I are expecting our first child in March, I was overwhelmed by the extent to which these non-renewable resources largely disappear within the span of only a single lifetime.

So, my design regrets.

If I’d left myself more time for working on this project, I would have animated the timeline and had the resources gradually fade out over time, versus just disappearing at specific points on the timeline. I also would have spent more time focusing on how to better represent the resources themselves – maybe with images, instead of text, or by focusing on the uses for the resources rather than the resources themselves.

Information is Beautiful posted the winning designs last week.  They’ll also be announcing their next design challenge any day, likely via their awards blog.  As a big advocate of both viewing nifty designs and playing with new projects, I’d recommend checking out them both.