Dries Buytaert

Drupal 7 usability: next steps

Improving the usability of Drupal is very important. Yesterday, I wrote about the progress that was made on Drupal 7 usability -- the community has made a ton of incremental improvements, while Mark and Leisa have been preparing mockups and wireframes that provide significant over-arching improvements to Drupal's ease of use. Combined, I believe these efforts could make Drupal 7 a great release. A release that the Drupal project needs since our competitors are catching up in terms of functionality and flexibility. Likewise we need to catch up in terms of design and usability. It is my belief that we can develop a user experience for our project that is game changing, and that completely resets people's expectations both for Drupal and our competitors.

As I wrote in yesterday's status update, an objective without a strategy remains a dream. We added a strategy by inviting Mark and Leisa to help but unfortunately, a strategy without execution remains a dream too. In other words, the time has come to start translating some of Mark and Leisa's work into working code. With 3 months until the Drupal 7 code freeze, it is time to build working prototypes so we can do user testing, refine the user experience, figure out the remaining design questions, and ultimately, get everything right for inclusion in Drupal 7.

Needless to say, this is a non-trivial amount of work that requires a highly collaborative process in what seems like a very limited time. It's easy to throw up our hands and say that it's impossible, that we should wait for Drupal 8, or whatnot. However, maintaining the status quo is simply not an option. Drupal 8 won't be used in production for, probably, at least another two years. As I wrote before, I don't want to compromise on Drupal 7's usability.

Few projects win by maintaining the status quo. Projects that win are those that go ahead and always improve. It is OK to be afraid about the work that is ahead. If you're not scared, you're not doing the right thing. Both winners and losers have fear, but all winners have faith. We can do this -- we've done it before.

We have to acknowledge that we arrived at a special point. The time has come to start working together even more but as a community, we're still learning how we can best work together on usability and design issues. Retrofitting a usability process into what has traditionally been an engineering process comes with some challenges and frustrations. Based on the progress that we have made recently, I'm confident that we'll figure this out, and that the process will feel completely natural in the long term.

Here is how I think about it in the short term, and how you can help.

Mark and Leisa will be helping us until the end of July to refine the current mockups based on more user testing, feedback from the community, and additional research -- they'll also be available to help answer questions and to help all of us sort through any implementation details. Their current mockups are a great starting point -- they'll force us to collaborate more and to streamline the process. The more mockups we can implement and the sooner we can implement them, the more testing Mark and Leisa can do, the more problems we'll discover, and the easier it becomes to refine things.

If you are a Drupal developer, look for issues tagged 'd7ux'. We're also using a tag called 'd7ux-design-question' to track non-engineering questions for Mark, Leisa and the usability team -- this should make it easier for everyone to stay on top of the progress. For people new to Drupal, Leisa and other people in the Drupal usability team launched a micro-projects initiative.

At this point, I've no plans to postpone the code freeze date. We started working on Drupal 7 in February 2008, so it has been a long release cycle already -- as it stands, Drupal 7 probably won't be released before January 2010. Instead of pushing back the code freeze date, I'd like to invite all of you to help with the implementation of the mockups that Mark and Leisa are providing. If you are a company, you can help by contributing some dedicated resources to help. If you're contributing as an individual, we appreciate every hour or day that you can put into this project. If you are a module maintainer, now would be a good time to figure out what this means for your module and how it could fit in. Get involved now, and help make Drupal 7 a game changing release!

To help, Acquia will start contributing a significant amount of its engineering resources. Starting this week, we'll contribute two or three people full-time until the end of the summer! Probably a combination of Gábor Hojtsy, David Rothstein, and Paul Lovvik. Jeff Noyes and Jason Reed will assist with user testing and design. Also, after talking to Mark Boulton, he has agreed to contribute one week a month of Tim Millwood's time (a Drupal developer that they have on staff) from now on to code freeze.

I hope others will step up to dedicate some of their time and resources as well -- when you do, make sure to announce it in the comments and on your blogs. Together, we should be able to make a lot of progress. The more people from the Drupal community that can jump in to help, the better Drupal 7 will be for us all. Now is the right time to bring vision, strategy, and execution together to achieve our objective. Fear not, Drupal 7 will be a game changing release.

Drupal 7 usability: status update

Based on a survey that I conducted last year, it was clear that one of the community's key goals is making Drupal easier to use. This is not really all that surprising. An easier to use Drupal means a Drupal that attracts more users, and therefore more potential contributors. It also means existing contributors spend less time doing custom work on their sites or developing training materials to help make Drupal palatable to their clients, and more time on making Drupal rock. And finally, it is strategically important for the future of the Drupal project to make our usability match the prowess of our flexible APIs; our competitors are working to build out their feature set to match Drupal's, and if we do not work to make Drupal's usability radically better, we risk falling victim to The Ockham's Razor Principle of Content Management Systems.

In this post, I want to give an update on the Drupal 7 usability work that has been done to date.

The good news is that thanks to the hard work of many people in the community, a lot of great Drupal usability improvements have already gone into Drupal 7. For example, we improved support for WYSIWYG editors, added more drag-and-drop, improved administration of permissions, added a default install profile which starts to configure some sensible defaults for Drupal as a publishing system, improved the password strength validator, redesigned and simplified various configuration pages, implemented "vertical tabs" on content creation and editing forms (see screenshot below), removed extraneous interface text, and much more.

The more important trend, however, is that after years of tireless evangelizing, the Drupal community is starting to value usability increasingly more. Today, the usability team is more involved in the patch review process than ever before -- for example, Bojhan, a usability expert active in Drupal's usability team, has even learned how to create patches so that he can help make simple changes. Another good example, is that in the Drupal 7 development cycle, we have raised money to do formal usability testing. These efforts illustrate a fundamental underlying desire to get better at usability, so I expect to see even more great things in the months to come.

All this is great work that helps address some of Drupal's fundamental problems; however, a lot of these improvements are incremental -- making incremental improvements is what the Drupal community excels at. Making the drastic over-arching improvements to Drupal's ease of use that we really need for Drupal to grow its audience requires one to step back, take a fresh look at the big picture, interview people, analyze problems, and propose more radical changes that incorporate piles of feedback and ideas. Needless to say, this is a long and expensive process, and not something we'd be able to get to using incremental changes. An objective without strategy remains a dream so as I announced in February 2009, I thought it would be right for Acquia to finance Mark Boulton and Leisa Reichelt to help the Drupal community with exactly this.

For about 3 months now, Mark and Leisa have been working with the Drupal usability group and the larger Drupal community to figure out how we can improve the usability of Drupal 7. I figured an update was in order.

Mark and Leisa's work is being done in an open, collaborative, and transparent way, involving everyone in the Drupal community. The work is happening in a number of locations, and the best place to keep track of it all is at D7UX.org. Mark and Leisa also often hang out with the usability team on #drupal-usability on irc.freenode.net, and even have been known to venture into #drupal and #drupal-dev.

Some of the activities to date have included some 'blue sky design workshops' with community members at DrupalCon DC, one on one interviews a range of members of the the Drupal community and many potential future Drupal users, asking you to share the Drupal administration modifications you're already made and others you'd like to see made. As the work has progressed they have shared work in progress on Flickr, shared ideas, process and prototypes in videos on YouTube, conducted user testing on those prototypes (and reviewed some of your findings through crowdsourced usability testing), and much more.

In addition to paying Mark and Leisa, Acquia has contributed some of its own internal engineering resources. Jeff Noyes, Jason Reed (both senior designers) and myself traveled to meet with Mark and Leisa in London -- we flew in Yoroy, a key contributor to Drupal usability, to attend as well. We locked ourselves in a room for two days, and discussed and validated early design prototypes. You can read a summary of the visit on Jeff Noyes's site. Returning home from London, I couldn't be happier, because the designs and prototypes started to live up my vision and expectations of how easy to use Drupal could be.

Since then, Mark and Leisa have revised many of their prototypes based on the feedback that was provided, and continues to be provided by the larger Drupal community. The latest versions are available from D7UX.org. They aren't finished yet, but Mark and Leisa will be helping us until the end of July. Expect more mockups, working prototypes and lots of user testing and iterations. Eventually, we should also have some style guides. If you're interested in participating (and the more people who participate, the better Drupal 7 will be), you can view a list of current activities as well as well as the project framework, where the current thinking around 14 separate topics is being centrally collected and discussed.

However, beautiful usability improvements are no good to anyone unless they have code behind them. In my next blog post, I'll write about the next steps, and how we can all help translate these mockups into working code.

Acquia featured in the Boston Globe

Acquia was featured in today's Boston Globe. The article, titled Start-up built on free software, talks about the growing interest in Open Source during tough economic times. The Boston Globe is a daily newspaper based in Boston, owned by The New York Times Company, with an average circulation of 350,000. Nice!

WoodWing and Acquia: moving from print to the web

A couple of days ago, Acquia announced a technology partnership with WoodWing. WoodWing's Enterprise 6 Content Publishing Platform now ships with Acquia Drupal as the default CMS, as well as a connector module. WoodWing's typical customers are traditional publishers with strong roots in print -- and with names like Time Inc, Cisco, Michelin, Citibank and many more, they have an impressive list of customers at that.

For most publishers, print continues to decline, and the web continues to grow. It is no surprise that their online strategy has become increasingly important for them. Many are in the process of optimizing their tools, processes and people for the web, rather than for print. It is a very disruptive change if you're a 100-year old organization that is optimized for print.

As publishers get settled on the web, they learn that it is not about broadcasting a message. On the web, it is about having a dialogue with your readers. As a publisher, you have to figure out how to turn audiences into participants because it opens the doors for better advertising and new monetization opportunities. Clearly, it doesn't suffice to copy-paste the content that you prepared for print into your web content management system.

As the web continues to evolve, readers will have a much richer experience online, and as a result, I believe print media will end up being a lead generation tool for the website, not the other way around as it often is today. Drupal's combination of content management and social software capabilities helps publishers to stay on top of this trend, and provides their writers and journalists the tools needed to be successful on the web.

But just like WoodWing's customers can benefit from Drupal, I think Drupal can learn a lot from WoodWing, and the issues that their customers have to deal with. As I write this blog post, I'm in Greece attending the WoodWing Publisher Conference, not only to educate WoodWing partners and customers about Drupal, but also to learn more about WoodWing's technology and challenges their customers are faced with. Like Drupal, WoodWing has a great community sharing both expertise and ideas. I really believe this partnership is a great match on many different levels.

Drupal.org redesign code sprints: update 3

Ever wanted to see the Media Lab at MIT? An almost mythical place where "...the future is lived, not imagined ... a world where radical technology advances are taken for granted, ..." and technology is designed "... for people to create a better future".

I've visited the Media Lab before and it is highly recommended. This is your chance.

On Friday, June 12th, we're conducting a one-day Drupal.org redesign sprint at the MIT Media Lab. Designed to bring home our kicking re-design for Drupal.org designed by Mark Boulton and Leisa Reichelt, all Drupal designers and developers who would like to contribute to theming our new home at drupal.org are welcome, invited, and encouraged to attend. Attendance is limited, so you must sign-up in advance.

Even if you're "just" a newbie, we'll make sure to have you ready for attendance by activating your infrastructure.drupal.org account, by creating your own hosted copy of drupal.org for your localized testing, and by training you in working within the redesign theme issue queue.

If you are interested in helping -- and I hope you are -- please sign-up to schedule your training by visiting http://groups.drupal.org/node/22036. Thanks!

Drupal site crawler project

For years now, people have been asking me how many Drupal sites there are. This is, of course, something of a moving target and I figured that the only way to answer that question was to count all the Drupal sites out there one by one. Three years ago, I was finally motivated enough to write a Drupal site crawler that looks over the millions of websites online to find those powered by Drupal. The crawler initially ran for about 3 months, and it returned a lot of Drupal sites. For each one it found, I then did some data-mining to extract the location of their hosting servers. I made a heatmap visualization of all the Drupal sites on a map of the world, and was able to start tracking Drupal's geographical growth patterns over time. As a bonus, the crawler also counted Joomla!, Mambo, and Wordpress sites, as well as a number of other open source content management systems - it is good data on Drupal and the competition.

Writing that crawler was a really fun project. It taught me a ton about how Drupal is used, where Drupal was growing, and how we compared to other content management systems. On the technical side, I learned a ton about scalability. Thanks to my engineering background and my work on Drupal, scalability issues weren't new to me, but writing a crawler that processes information from billions of pages on the web, is a whole different ball park. At various different stages of the project and index sizes, some of the crawler’s essential algorithms and data structures got seriously bogged down. I made a lot of trade-offs between scalability, performance and resource usage. In addition to the scalability issues, you also have to learn to deal with massive sites, dynamic pages, wild-card domain names, rate limiting and politeness, cycles in the page/site graph, discovery of new sites, etc.

Even though the crawler ran for many months, I never really launched it or talked about it publicly. I personally lacked the resources (both my time and the money to run the servers) to keep it running all the time. I ultimately stopped the crawler altogether and to put the project on hold. Since then, a couple of things have changed: I learned a lot more about scalability thanks to my ongoing work on Drupal and on Mollom, which now processes hundreds of thousands of spam messages a day. I also co-founded Acquia along the way. Acquia shares my personal interest in tracking Drupal's growth and has the resources to help me revive my crawler. Last but not least, there has been a lot of innovation and knowledge sharing around how to solve scalability problems. As such, I'd now like to pick up the project where I left off 3 years ago, and relaunch it under the Acquia umbrella.

One thing hasn’t changed: the day remains +/- 24 hours long. I still don't have enough time to work on this, and have many priorities I didn’t have 3 years ago. That is why I'm looking for a summer intern who wants to take the lead on this project for a few months. The crawler is written in Java, so I'm looking for a student proficient in Java, and who also understands fundamental internet protocols such as DNS, HTTP and HTML. Experience with building scalable multi-server systems is a plus but not strictly required. I don't care where you live, but I only want to work with people who work hard and who have strong programming skills. Are you interested in being my intern for the summer? Contact me at my contact form and send me a copy of your resume. Or if you know someone who would be a good candidate, please point them to this blog post.

Buzzr, Acquia Gardens and D7UX

Earlier this week, Lullabot showed a preview of Buzzr (previously announced as "Codename"), a hosted version of Drupal 6 with a new user interface and user experience. Interestingly enough, the new user interface has remarkable similarities with the work that Mark Boulton and Leisa Reichelt have been doing for Drupal 7 (D7UX). It is my guess that the growing momentum around Mark's work encouraged Lullabot to give us a sneak preview behind the scenes of what was brewing since they first announced Codename. And from what I saw, Buzzr looks great!

Because the goal of Buzzr is to help make Drupal usable, some people asked me if there is a conflict with the work of Mark Boulton and the Drupal community, and the work we have been doing at Acquia with Acquia Gardens, which we'll build on top of the Drupal 7 usability work. I wanted to address some of these questions in this post.

First, all of us in the community have always shared a common goal. That shared goal is to make Drupal better. I think it is great when different groups of people independently arrive at a shared vision of how Drupal could become more usable. It implies that the approach embodies a lot of common sense.

Second, and while I can't speak for Lullabot, it is my hope that Lullabot will join the D7UX project to help advance the entire community. Their Buzzr work is based on Drupal 6, which might or might not make it easy to contribute.

At Acquia, we decided against building Acquia Gardens on Drupal 6. Our goal is to introduce Drupal to millions of new users so we decided to invest in Drupal 7 up front so we can stay really close to Drupal, create a tight feedback loop, help promote Drupal "as it is" and eventually, worry a bit less about upgrades while creating the best experience for new users. We want the Drupal project at large to win, despite some of the extra work, complexity and challenges such a strategy brings.

Either way, as a community, we can learn from Buzzr and identify what we like and dislike, what works well and what doesn't. Everyone, including Lullabot, benefits from that kind of feedback. So in all, it is great to see Lullabot provide more visibility into their work, and to contribute even more to Drupal. In many different ways, it paves the path to even more Drupal 7 usability improvements. In fact, Mark and Leisa have already been talking with members of the Buzzr team.

So if you ask me, I'm eager to see the exciting changes for Drupal that can result from the combined learning and work of the Lullabots, Mark and Leisa, Acquia and many of the other great members of the Drupal community.