First results from usability testing

Two weeks ago, we conducted formal usability testing for Drupal at the University of Minnesota Office for Information Technology's usability lab. As I wrote earlier, the university has a professional usability lab that allowed us to record eye-tracking data and video.

Eye tracking hot spot map

The heat map shows where the users look the first 5 seconds after landing on Drupal's main administration page. The red X's show where the users clicked.

The blue dot in the video shows what the user is looking at on Drupal's user account creation page. Several users backed out in a panic because they thought they caused an error condition due to the red text.

Seeing users consistently fail at what we consider to be basic tasks (e.g. creating a new content type or adding a new user account) is a true eye opener. Let's be clear about this: this is Drupal's fault, not the users' fault. The good news is that we came out of this with a long list of usability problems that we can fix.

In fact, the Drupal team that participated in Minnesota tried to process much of the data prior to our DrupalCon Boston presentation on usability testing: check out our preliminary report from the usability testing (PDF, 6MB) or participate in the Drupal usability group where more information and results will be made available.

Massive thanks to Chad and Cody of the University of Minnesota for making this possible, and a big thumbs up for all the Drupal participants that are committed to making Drupal easier to use. Let's start addressing those problems!

Comments

Alan F. (not verified):

Is there any chance to find those quicktime movies in some other format (for example FLV)?

March 10, 2008 - 11:55
Said Bakr (not verified):

I agree with Alan F, videos is going to be better and usable if it was in flv format.

March 13, 2008 - 20:42
bertboerland (not verified):

Great stuff Dries. And for those who want videos in another format, ffmpeg is your friend.

March 10, 2008 - 12:07
Dominik (not verified):

Great stuff confirming much of my own experience with users. Sorry to have missed the presentation.

Two related points:

  1. I don't think the situation around the creation of new content is as dire as the results suggest. I've been able to get a lot of people up to speed on this very quickly. The wording of the first task was, at least, a part of the problem because it triggered the context of form submission which I think for most web developers is different from content creation. Something more neutral such as 'Find a way for users to add workshops to the site' would have probably made it easier to figure out. I was confused about the form bit for a moment myself (and I've created hundreds of content types as well as forms).
  2. What's not taken into account is forms of learning. Once things are explained, users find Drupal very easy to use (by and large). But the problem is that they very often don't want to put in the personal learning time. I've sent many users to some of the great screencasts and handbook pages out there but they ended waiting for me to come and explain it to them. So, what I'm trying to say is that there will always be a barrier to entry created by real-world factors. But a series of 3-5 minute professionally designed How-To videos to go at the very top of the Drupal manual pages (and maybe one to go to the front-page) would take things a long way.
March 10, 2008 - 13:21
Benjamin Melançon (not verified):

People interested in getting funding or helping others get funding for contributing usability enhancements to core based on usability lab studies should join the Knight Foundation Drupal Group.

This initiative that the Knight Foundation announced at DrupalCon is about coming up with a community-driven way (that means us) to decide what projects to fund, as well as funding Drupal projects that will help public interest journalism and local communities through digital technology.

March 10, 2008 - 13:57
Jordi Bufí (not verified):

Hi, I couldn't see the videos so I uploaded them to Youtube:

http://www.youtube.com/watch?v=J3kTdlV3pZs

http://www.youtube.com/watch?v=Xkl2czgOBC4

I hope that this won't bother you.

March 10, 2008 - 15:19
tjholowaychuk (not verified):

Wow that must be interesting to see all the eye tracking that was captured, pretty neat stuff!

March 10, 2008 - 15:25
Zohar (not verified):

The blue dot hysterical movie is a killer! Imagine a young terrified student in front of the screen, especially in the password strength test... LOL

March 10, 2008 - 16:29
Robert Douglass (not verified):

To be fair, the person in the video ended up choosing a more secure password, which is what we wanted. Furthermore, they apparently did so after reading (and carefully considering) the help text.

March 11, 2008 - 11:44
gaele (not verified):

The blue dot in the video shows what the user is looking at on Drupal's user account creation page. Several users backed out in a panic because they thought they caused an error condition due to the red text.

There's an issue for this: http://drupal.org/node/199870

March 11, 2008 - 14:21
As If (not verified):

Let's be clear about this: this is Drupal's fault, not the users' fault.

While that's true, it's also not surprising to ye olde Drupal developers. The first question I am asked by almost every client is "Can you make it NOT look like Drupal?" And the creation of an "owner" role for the site owner (thereby hiding most of the complexity of the admin interface) is part of almost every commercial Drupal job. It's like construction scaffolding that folds up into the wall after you're done (just in case you ever need to re-engineer your apartment)!

So I guess I'm saying there is a bright side - developers get plenty of work "masking" various elements of the Drupal interface. I prefer having that flexibility to not having it.

March 14, 2008 - 07:11
Anonymous (not verified):

a "content admin" and a "site admin" role should come with Drupal "out of the box" because almost every installation creates one and modules should be required to set default permissions for these roles, so that enabling a module does not automatically require a user to visit the permissions page to enable a bunch of permissions for that module!!

July 3, 2008 - 18:15
Mike Schinkel (not verified):

Let's be clear about this: this is Drupal's fault, not the users' fault.

Kudo's for taking the leadership role on recognizing that users are never the ones to fault for poor usability. The fact that you participated in this usability study gives me great hopes for the future of Drupal!

June 29, 2008 - 02:17

Add new comment

© 1999-2014 Dries Buytaert Creative Commons Attribution-NonCommercial-ShareAlike 3.0 License.
Drupal is a Registered Trademark of Dries Buytaert.