Dries Buytaert

Drupal 7 testing: status update and next steps

The first version of Drupal's automated test framework was developed in 2004 by Moshe Weitzman. It started as a simple wrapper around the SimpleTest unit testing framework. Later, Thomas Ilsche and Rok Zlender extended it as part of the Google Summer of Code projects of 2005 and 2006. NowPublic and others continued to sponsor Rok's work into 2008. Today, Jimmy Berry is the principal contributor of the Drupal test framework, as well as the main developer and maintainer of Drupal.org's automated test infrastructure. Behind the scenes, Kieran Lal was instrumental in helping to ensure our test framework received financial support, project management, hardware resources, and server administrators.

In the February 2008, I declared that Drupal should embrace test-driven development, and shortly thereafter, we added the test framework to Drupal 7 core. In addition to a test framework for core, Jimmy Berry and Chad Phillips developed a second generation automated testing framework for drupal.org -- so far, it has tested over 8500 unique patches on Drupal.org.

Yesterday, Jimmy announced that he will be working part-time as an intern for Acquia this summer. At Acquia, we like what Jimmy is doing, and because he has been looking for a financial sponsor for a while now, I decided Acquia should step up and help Jimmy for part of the summer. Frankly put, Jimmy's work has the potential of doubling the velocity of our developer community, and that is well worth sponsoring. Jimmy will spend his time with Acquia to continue his work on Drupal 7's test framework and the automated test infrastructure for drupal.org.

For well over a year now, core development has used a test-driven development strategy combined with automated testing. I think all core developers working on Drupal 7 will unanimously agree when I say that our test infrastructure has drastically improved our velocity and effectiveness. Overall, patches get accepted faster. The automated tests allow us to focus on the architectural and the algorithmic changes introduced by a patch, rather than having to worry about unexpected side-effects. This helps both patch reviewers and core developers contributing patches. Furthermore, thanks to the tests, we have a much better feel about the stability and the health of Drupal 7. I am optimistic that our code freeze period for Drupal 7 could be shorter than prior releases. As the project lead, our test framework helps me sleep better at night. The stability and health of our code base is important to me, but frankly, it is at least as important for the many Drupal users, or for those people and organizations looking to adopt Drupal.

By adopting a test-driven development strategy we raised the bar for core development, and I strongly believe we have to do the same for contributed modules. While the community cannot force module maintainers to write tests for their modules, I do want to support those that do write tests. The best way to support them, is to provide them the same tools and infrastructure that we use for core development. And as the maintainer of a contributed module myself (i.e. the Mollom module), I can't wait to have the same tools available that I have available when working on Drupal core.

Jimmy is working on making drupal.org's automated test infrastructure available to contributed modules, integrating the test results in the project pages as an incentive tool, improving the documentation to help people get started, and closing some functional gaps such as the ability to test Drupal's installer and upgrade system. We're also working on deploying the third generation of the automated testing framework. This new version will make it easier to add testing servers so we can triple the number of servers needed to run tests, and start testing the non-MySQL backends that are supported by Drupal 7.

All in all, I'm very optimistic that we'll succeed in our goal to adopt a test-driven development strategy.

Usability work at DrupalCon DC

I recently announced that one of the ways in which Acquia is contributing to Drupal is by providing funds for Mark Boulton and Leisa Reichelt to help the Drupal community improve usability in Drupal 7. Mark and Leisa are beginning their work at DCDC this week!

One of the things they're hosting, with the help of Jeff Noyes, are "Blue Sky Design Workshops". The workshops are Wednesday 3-4pm and Thursday 9-10am and 11:30-12:30am. The "Blue Sky" part of the title notes that these sessions are designed to push the boundaries of usability in Drupal, to get the community together, to brainstorm about new possibilities and to dream up solutions.

Furthermore, the Drupal usability team has just completed a second round of formal usability testing at the University of Baltimore and I'm really looking forward to learn more about the results of their work.

To keep up the involvement after DCDC, or if you're following along from home, consider joining this Mark Boulton D7 UX group as well as the regular usability group on groups.drupal.org for updates. Make sure to tune in!

Usability is one of those areas where we can really break new ground with our next release. We've already done much work, but have more to do. Through activities like this, and with the feedback from some "fresh eyes" through research and user testing, I'm very hopeful and excited about where we're headed.

Usability, usability, and usability

The Interaction Design and Information Architecture program at the University of Baltimore and a team of eight graduate students have completed a usability study on Drupal. The result is a great report (PDF) and an incredibly valuable video which they shared on drupal.org. It is too important not to share, so the video is also embedded below.

The results are consistent with the results from usability tests done at the University of Minnesota.

The results can't be ignored.

I printed the report, taped it on my wall, and I won't release Drupal 7 until I crossed of at least 90% of the problems they identified.

We have a test framework in Drupal 7

Every major Drupal release we should ask ourselves what steps we can take to double the capacity of our community.

I spent the weekend in Paris where we had a two day code sprint. Our main accomplishment was getting Drupal's test framework into Drupal core -- the culmination of three years of hard work carried out by many people and companies in the Drupal community.

Here is why a strong investment in testing will help double the capacity of our community:

• For developers, upgrading, maintaining and releasing your modules becomes easier. The combination of test results and code coverage reports makes it easier to determine the release readiness of your code. This translates to fewer betas, shorter code freeze periods and more frequent releases. Furthermore, design for testability leads to easier to use and more complete APIs. It is guaranteed to make Drupal a better development platform.
• For end users, it is important that we provide quantifiable reporting on the health of Drupal core and the many contributed modules.
• For patch reviewers, tests are great because it allows them to focus on the architectural and the algorithmic changes that the patch introduces. With good test coverage, we can rely on the tests to discover any unwanted side-effects. Patches can be committed faster.
• For people new to Drupal, tests lower the barrier to entry and encourage collaboration and innovation, two of Drupal's core values. With good test coverage, you don't necessarily have to understand the entire code base before you can comfortably propose changes or help maintain a module.
• For me, it means I'll sleep better at night. ;-) With hundreds of thousands of people using Drupal, the availability of a test framework takes some pressure of my shoulders.

As of today, it is expected that you submit test cases with your patches for Drupal 7. Writing good tests takes time: it is not unlikely that you'll spent twice as long working on a patch. This might take some time getting used to but you'll find that it pays off and that it is actually very rewarding.

Thanks to Ori Pekelman of AF83 for being such a great host when in Paris. And a special thanks to everyone who helped get the test framework in Drupal 7.

First results from usability testing

Two weeks ago, we conducted formal usability testing for Drupal at the University of Minnesota Office for Information Technology's usability lab. As I wrote earlier, the university has a professional usability lab that allowed us to record eye-tracking data and video.

The heat map shows where the users look the first 5 seconds after landing on Drupal's main administration page. The red X's show where the users clicked.

Seeing users consistently fail at what we consider to be basic tasks (e.g. creating a new content type or adding a new user account) is a true eye opener. Let's be clear about this: this is Drupal's fault, not the users' fault. The good news is that we came out of this with a long list of usability problems that we can fix.

In fact, the Drupal team that participated in Minnesota tried to process much of the data prior to our DrupalCon Boston presentation on usability testing: check out our preliminary report from the usability testing (PDF, 6MB) or participate in the Drupal usability group where more information and results will be made available.

Massive thanks to Chad and Cody of the University of Minnesota for making this possible, and a big thumbs up for all the Drupal participants that are committed to making Drupal easier to use. Let's start addressing those problems!

Embracing test-driven development

At DrupalCon Barcelona about 20 people gathered together for the Quality Assurance Birds of a Feather session. This led to the development of a continuous testing infrastructure for Drupal. People that want to write tests for Drupal can, but don't have to.

Second, in my State of Drupal presentation at DrupalCon Barcelona, I shared a list of the top-10 most requested Drupal improvements. On spot 9 we had "better internal APIs" and on spot 10 we had "better external APIs (import/export, web services)". APIs that are designed for testability, are often better APIs. This comes from the fact that testability leads to more cohesive code with loose coupling. Important for readable, agile code.

Third, Drupal 6 has been in a code freeze for 6 month. Often times, the behavior introduced by one patch inadvertently breaks the behavior of existing functionality. In absence of tests, it takes a while to shake out all the bugs and before you're actually confident to roll a new major Drupal release.

As a result, people in the community have suggested to put more emphasis on testing in Drupal. Writing tests is expensive so it is hard to model the cost and benefits. However, the only way to past the costs and to experience the true benefits, is to take testing to the extreme. There is no middle ground. If we want to be serious about testing, we need to enforce, not just encourage, testing in the Drupal 7 development cycle and beyond. (And maybe we should consider introducing tests for Drupal 5 and Drupal 6 maintenance releases.)

In such a world, patches that fix bugs should come with a test case that demonstrates the behavior being fixed. That patch would then be added to the regression test suite and run continuously to avoid that the bug gets re-introduced. In addition, patches that add new features or enhancements should come with test cases for this new functionality. Everything should be tested.

Of course, this would require a significant commitment from the whole Drupal developer community. There is bound to be some controversy about the need to learn how to use Drupal's test framework and the overhead and complications of having to write and submit a test. But over time, fewer people will be frustrated by bugs, the burden on patch reviewers will go down, code freezes will become shorter, APIs will continuously improve, and we would have more confidence in the stability of our releases. Last but not least, it actually improves overal productivity and encourages change and experimentation -- something the Drupal community really takes pride in.

As we learned before: sweeping changes are required to make major advances in technology, and often times there is a lot of pain before the pay-off. Are we willing to take some pain? Are we willing to enforce tests to be written as part of the Drupal 7 development process? I'm willing to fully embrace testing, but only if you are willing to write tests ...

Drupal usability testing

When I predicted that there would be a big and concentrated effort to further improve Drupal's ease of use in 2008, I was cheating a bit ... The past months we've been preparing some formal usability testing for Drupal that will be conducted at the University of Minnesota Office for Information Technology's usability lab. The university has a professional usability lab that will allow us to record eye-tracking data and video which will be provided to the Drupal community. Très cool!

Chad Fennel and Cody Hanson of the University of Minnesota Libraries have taken an incredible lead in this; they secured the lab, they prepared test scenarios, they recruited test users, they are coaching us how to do usability research, and last but not least, they are footing part of the bills. Needless to say, this is a significant contribution to the Drupal project -- and hopefully one that will have big impact on Drupal 7.

Needless to say, formal usability testing is only one way to make Drupal easier of use. There are a lot of usability findings out there already (example 1, example 2), and many of us know first-hand which changes need to occur to improve Drupal's usability. All it takes is a knack for usability, some development skills and a lot of time and effort. Ultimately, everyone can help -- with or without a professional usability lab.

The past years, I've pushed hard to make usability reviews part of the development and patch review process. I strongly believe that usability reviews should be tightly integrated with the software development process; rapid usability feedback and incremental usability improvements allows us to build better software faster. And it is working -- every major Drupal release has become significantly easier to use.

Nonetheless, there is a lot of work left to be done. I hope that the formal usability testing at the University of Minnesota Libraries will provide us with a fresh perspective and that it helps train the Drupal community's eye for usability. Because more than anything else, I want to us to flatten the Drupal learning curve.

More information about the usability testing will be made available shortly. The goal is to share some first results at DrupalCon Boston next month.