First results from usability testing

Two weeks ago, we conducted formal usability testing for Drupal at the University of Minnesota Office for Information Technology's usability lab. As I wrote earlier, the university has a professional usability lab that allowed us to record eye-tracking data and video.

Eye tracking hot spot map
The heat map shows where the users look the first 5 seconds after landing on Drupal's main administration page. The red X's show where the users clicked.

The blue dot in the video shows what the user is looking at on Drupal's user account creation page. Several users backed out in a panic because they thought they caused an error condition due to the red text.

Seeing users consistently fail at what we consider to be basic tasks (e.g. creating a new content type or adding a new user account) is a true eye opener. Let's be clear about this: this is Drupal's fault, not the users' fault. The good news is that we came out of this with a long list of usability problems that we can fix.

In fact, the Drupal team that participated in Minnesota tried to process much of the data prior to our DrupalCon Boston presentation on usability testing: check out our preliminary report from the usability testing (PDF, 6MB) or participate in the Drupal usability group where more information and results will be made available.

Massive thanks to Chad and Cody of the University of Minnesota for making this possible, and a big thumbs up for all the Drupal participants that are committed to making Drupal easier to use. Let's start addressing those problems!

Comments

Dominik (not verified):

Great stuff confirming much of my own experience with users. Sorry to have missed the presentation.

Two related points:

  1. I don't think the situation around the creation of new content is as dire as the results suggest. I've been able to get a lot of people up to speed on this very quickly. The wording of the first task was, at least, a part of the problem because it triggered the context of form submission which I think for most web developers is different from content creation. Something more neutral such as 'Find a way for users to add workshops to the site' would have probably made it easier to figure out. I was confused about the form bit for a moment myself (and I've created hundreds of content types as well as forms).
  2. What's not taken into account is forms of learning. Once things are explained, users find Drupal very easy to use (by and large). But the problem is that they very often don't want to put in the personal learning time. I've sent many users to some of the great screencasts and handbook pages out there but they ended waiting for me to come and explain it to them. So, what I'm trying to say is that there will always be a barrier to entry created by real-world factors. But a series of 3-5 minute professionally designed How-To videos to go at the very top of the Drupal manual pages (and maybe one to go to the front-page) would take things a long way.
Benjamin Melançon (not verified):

People interested in getting funding or helping others get funding for contributing usability enhancements to core based on usability lab studies should join the Knight Foundation Drupal Group.

This initiative that the Knight Foundation announced at DrupalCon is about coming up with a community-driven way (that means us) to decide what projects to fund, as well as funding Drupal projects that will help public interest journalism and local communities through digital technology.

Zohar (not verified):

The blue dot hysterical movie is a killer! Imagine a young terrified student in front of the screen, especially in the password strength test... LOL

Robert Douglass (not verified):

To be fair, the person in the video ended up choosing a more secure password, which is what we wanted. Furthermore, they apparently did so after reading (and carefully considering) the help text.

As If (not verified):

Let's be clear about this: this is Drupal's fault, not the users' fault.

While that's true, it's also not surprising to ye olde Drupal developers. The first question I am asked by almost every client is "Can you make it NOT look like Drupal?" And the creation of an "owner" role for the site owner (thereby hiding most of the complexity of the admin interface) is part of almost every commercial Drupal job. It's like construction scaffolding that folds up into the wall after you're done (just in case you ever need to re-engineer your apartment)!

So I guess I'm saying there is a bright side - developers get plenty of work "masking" various elements of the Drupal interface. I prefer having that flexibility to not having it.

Anonymous (not verified):

a "content admin" and a "site admin" role should come with Drupal "out of the box" because almost every installation creates one and modules should be required to set default permissions for these roles, so that enabling a module does not automatically require a user to visit the permissions page to enable a bunch of permissions for that module!!

Mike Schinkel (not verified):

Let's be clear about this: this is Drupal's fault, not the users' fault.

Kudo's for taking the leadership role on recognizing that users are never the ones to fault for poor usability. The fact that you participated in this usability study gives me great hopes for the future of Drupal!