Embracing test-driven development

At DrupalCon Barcelona about 20 people gathered together for the Quality Assurance Birds of a Feather session. This led to the development of a continuous testing infrastructure for Drupal. People that want to write tests for Drupal can, but don't have to.

Second, in my State of Drupal presentation at DrupalCon Barcelona, I shared a list of the top-10 most requested Drupal improvements. On spot 9 we had "better internal APIs" and on spot 10 we had "better external APIs (import/export, web services)". APIs that are designed for testability, are often better APIs. This comes from the fact that testability leads to more cohesive code with loose coupling. Important for readable, agile code.

Third, Drupal 6 has been in a code freeze for 6 month. Often times, the behavior introduced by one patch inadvertently breaks the behavior of existing functionality. In absence of tests, it takes a while to shake out all the bugs and before you're actually confident to roll a new major Drupal release.

As a result, people in the community have suggested to put more emphasis on testing in Drupal. Writing tests is expensive so it is hard to model the cost and benefits. However, the only way to past the costs and to experience the true benefits, is to take testing to the extreme. There is no middle ground. If we want to be serious about testing, we need to enforce, not just encourage, testing in the Drupal 7 development cycle and beyond. (And maybe we should consider introducing tests for Drupal 5 and Drupal 6 maintenance releases.)

In such a world, patches that fix bugs should come with a test case that demonstrates the behavior being fixed. That patch would then be added to the regression test suite and run continuously to avoid that the bug gets re-introduced. In addition, patches that add new features or enhancements should come with test cases for this new functionality. Everything should be tested.

Of course, this would require a significant commitment from the whole Drupal developer community. There is bound to be some controversy about the need to learn how to use Drupal's test framework and the overhead and complications of having to write and submit a test. But over time, fewer people will be frustrated by bugs, the burden on patch reviewers will go down, code freezes will become shorter, APIs will continuously improve, and we would have more confidence in the stability of our releases. Last but not least, it actually improves overal productivity and encourages change and experimentation -- something the Drupal community really takes pride in.

As we learned before: sweeping changes are required to make major advances in technology, and often times there is a lot of pain before the pay-off. Are we willing to take some pain? Are we willing to enforce tests to be written as part of the Drupal 7 development process? I'm willing to fully embrace testing, but only if you are willing to write tests ...

Comments

Philippe Jadin (not verified):

A middle ground (or first phase) could be that patch provided with tests would have another status in the issue queue and be more likely to be committed

It could be something like

- active (test provided)
- patch (test passed)

A bug report with a test provided would be a "proof" that there is a problem. The same test could be used to check that the patch provided later does in fact fix the issue

February 15, 2008 - 09:28
catch (not verified):

Things that takes most human time where there's fewer reseurces are 1. upgrade paths 2. postgres testing (often part of 1). Since you can't simply patch p0 < patch and click for those.

I'd like to see 'click through core' and 'upgrade major version' tests running on most relevant patches - that'd certainly save some time, and those tests would be reusable over a lot of patches. Requiring a test with every patch would IMO make the contributors learning curve a lot higher than it is now. New statuses sounds great as an intermediate step though.

February 15, 2008 - 12:23
greggles (not verified):

This seems like a great idea to me and it helps us slowly move into that phase without making a full, harsh, top-down commitment.

Lots of folks are blocked from contributing because of lack of knowledge - PHP, CVS, Issue Queues, English(!) - these prevent people from being able to help. Adding in the "must have a test" hurdle will keep even more people away.

Part of the solution will be specialists. Much like chx/pwolanin handle most menu bugs or eaton tends to do lots with node rendering and FAPI, or keith.smith tends to do a lot of string improvement, and catch does a million reviews, we need to get some folks who are really capable and speedy with developing tests so that they can get involved in a patch that looks great except for lacking a test. I think a transition period like you describe will help that.

I don't think this should run too long, though. Perhaps just for a few months and then we start making tests required for core.

Now, to go see about writing some simpletests for my contributed modules... ;)

February 15, 2008 - 12:26
Jeroen (not verified):

I think Drupal has to be taken serious. It's not something you want people with "lack of knowledge" contributing to. If one feels that writing a test is an extra hurdle to overcome then he/she is not fit for contributing to software that runs many prominent websites.

February 15, 2008 - 13:24
catch (not verified):

Actually people with 'lack of knowledge' are exactly the sort of people we want contributing to Drupal core - 'lack of knowledge' is a great asset for reviewing usability changes and improving documentation amongst other things, and n00bs don't stay such for long.

February 15, 2008 - 14:04
greggles (not verified):

According to the data I analyzed about Drupal6 contributers, 71% of the patches to Drupal6 were created by people who only had 3 or fewer patches integrated into core. 52% of the patches were the only patch for the person.

I think it is a reasonable assumption that people who contributed only 3 patches (and moreso only 1 patch) are more casual developers who are more likely to have a hard time with something like simpletests.

So, if you want to exclude those people I think you will find that Drupal core suddenly becomes less exciting of a place for development...

I know that Dries is aware of this (perhaps moreso than most people are) and that it is why he is promoting this idea slowly and in a way which will hopefully keep those casual developers involved in the process.

February 15, 2008 - 22:01
cwgordon7 (not verified):

How do you calculate that? According to the same dataset I get a different result...

Also, speaking as one of those with 2 patches, I can say that I'm perfectly comfortable with SimpleTest... ;)

June 2, 2008 - 21:02
Yves Hanoulle (not verified):

the more people that commit that don't know most of the code, the more tests are important.
You make it sound like tests are hard to write.
Adding a test after the fast is hard. Doing TDD (or BDD) actually makes it easier to write tests.

Yes the core drupal developers need to spend some time making it easier to add tests to core parts.

February 11, 2011 - 22:34
Anonymous (not verified):

Oh boy ... maybe "you" should focus on the important things the "users" (+dev) need most: Views and CCK into core ... Then start the interesting development stuff. I know PHP is mostly boring to program :-)

My 2 cents

February 15, 2008 - 09:43
Robert Douglass (not verified):

@ Anonymous of the 2 Cents: in my opinion expanding the test coverage of Drupal and providing the infrastructure, culture, workflow and discipline to integrate testing into Drupal is more important than any feature request, including the ever popular quest for CCK and Views-like capabilities in core.

Drupal (and most software) is plagued not only with bugs, but also with code that is easy to break. Having a testing culture that is central to the philosophy of software development will make it *easier* and *faster* to get new exciting features into core.

Once again Dries shows his leadership and we can see here why Drupal is so great; because Dries knows what's important.

Testing is officially a #1 priority for Drupal. It is up to the community to now make it real. There is no doubt in my mind that this is the right direction for us, and that investing time and effort into testing will pay off with huge dividends in the long run.

February 15, 2008 - 12:25
dvessel (not verified):

Agreed. With the exception of catch and maybe a few others, there really aren't many reviewers. So, moving the focus away from individual reviewers to solid test cases looks like the right step. I never dealt with unit testing but anything that helps all of us create a more solid framework I'm all for. In the end, we'll all be stronger for it.

Now, on to figuring out what it's all about. Hope to see more documentation on this. :-)

February 15, 2008 - 15:07
dvessel (not verified):

Oh, there's a lot on this already linked from sir Douglass's site.

http://robshouse.net/blog-post/testing-now-official-drupal-priority

February 15, 2008 - 15:41
catch (not verified):

Not to mention that 'views and cck into core' is very much 'interesting development stuff' that's likely to occupy much of this release cycle.

February 15, 2008 - 14:05
chx (not verified):

Here is an idea. If someone, say, submits a nice UI improvement which breaks some simpletests (because the form changed) then we still commit that but open a critical bug saying "simpletest X is broken" and then fix that. You can be sure I will be forefront of the people who pay attention to these bugs and fix them.

February 15, 2008 - 14:30
Cosmic (not verified):

IMO, the Coder module is a good start. By using it on a newly downloaded Drupal 6.0 release I was able to find many critical errors with the core modules and code. If we were to continually file critical bugs for each of the critical bugs we find using Coder and made it requirement that new patches should not create new Coder-identified critical bugs then we would eventually reduce the number of critical bugs to zero.

February 15, 2008 - 15:26
dhubler (not verified):

Tests are kinda hard to write for Drupal. So this does increase the learning curve. DruapalTestCase hides some of it but there is an endless amount of work here. Db needs to have seed data, db needs to have empty db, etc. I think there needs to be advancements in the tools to make writing tests like falling off a log.

Once there is a critical mass of tests, and people get saved a few times from making a serious bug because a unit test caught something, i have confidence testing will catch like wildfire.

February 15, 2008 - 18:12
Anonymous (not verified):

dhubler, there is no need for any special state of the database. You can change everything as it is needed and the code automatically reverts at the end of the test. They are very easy to write, there is even a simpletest_automator project which will take the a first cut at the test.

February 15, 2008 - 21:51
dhubler (not verified):

I was basing this on looking at samples in simpletest module in "tests" directory and what info I could find in d.o and g.d.o. Are there more example somewhere?

I envision at least 3 different sets of test.

  • No db or web access, extends SimpleTestCase
  • Db or web access, extends DrupalTestCase
  • Web-script like tests (e.g. Selenium) that can perform xpath expressions validation, test ajax results, number of windows active, etc.

Then I can write focused tests like

test_blog_entry() {
$actual = create_blog("tweet", 'birds like to tweet');
$this->assertEquals($actual['name'], 'tweet');
$this->assertEquals($actual['body'], 'birds like to tweet');
}

and I wouldn't have to create a user, give auth permssions, enable/disable modules (unless of course you actually wanted to test that).

Nothing stops me from writing tests for #1 and #3 except maybe the fact no one has started to or committed them into CVS to get this started. Personally I find #1 and #3 more useful because they do not contain drupal specific testing API. Just the code I would be creating or knowledge of drupal web interface.

All, I appreciate your expertise on this, I'm new to Drupal/PHP.

February 15, 2008 - 23:12
dragonwize (not verified):

I think chx has a good idea on this topic. As Dries says, testing will be most useful when required. However, the learning curve does grow. Not requiring the same person who writes a patch to write the test is a nice balance.

February 17, 2008 - 07:46
Dries:

A possible solution is to enforce patches from long-term contributors but to be more forgiving for people that are new to Drupal development.

February 17, 2008 - 21:41
vladimir.dolgopolov (not verified):

There are my considerations:

1. It is necessary to make "The Uses Cases Library". A place for developers they could take a couple of uses cases for testing their modules to work.

2. A tool for recording user actions on Drupal site (Selenium?).

3. The developer should write tests as easy as write comments.

4. Tests will be written both module's author and other users.

5. To make on each module ability to post a test case (as it is made with issues, like this: http://drupal.org/project/tests/my_module)

6. There should be modules raining based on quantity and quality of tests.

7. There is not always can restore a system state after the tests. Simpletest tries to restore it, but it's better to have isolated recreated testing sandbox.

8. The test should be stored in the separate file.

9. There should be an estimation of quality of the test on the code coverage (XDebug can help).

February 18, 2008 - 05:12
Philippe Jadin (not verified):

And don't forget that tests are not the holy grail of bug free software. It's quite easy to write the wrong test and have a module that pass it altough it's buggy.

I'm not sure, but I have the feeling that writing quality, "to the point" tests is even harder than writing good, "bug free" code. I'm not sure a newbie can always write a usefull test.

That's why adding some more status to the issue queue allows someone to report a bug, someone else to create the text, and again someone else to provide the patch.

I don't think it's a good idea to enforce everyone to do everything in the (sometimes) long life of a bug.

my 0.02

February 18, 2008 - 10:13
Dude (not verified):

I'm strongly against requiring tests. If a developer wants to write them, let them. But a newbie developer should not have to write tests. It's already harder to develop for the 6th version, and now you want to introduce mandatory testing? I fear that the 7th version of Drupal will be when it jumps the shark. And why is it that every Drupal version has to break backward compatability? Are you trying to outdo Microsoft?

February 19, 2008 - 03:51
Jeroen (not verified):

As a developer at a company with over 20 000 tests for 1 software product (unit tests / component tests / virtual and physical tests) I can say that testing is an absolute necessity. When approaching people interested in Drupal for their company's website I can imagine them asking about the tests or test-framework in place. Drupal, the core at least, isn't a hobby anymore I think and it needs stability. The surrounding modules are a different thing.

So no, new developers shouldn't have to write tests for their modules as they'll not contribute to core anyway. But I think it'd be good to have a set of regression tests for Drupal core, run on Drupal instances with and without modules enabled. This will cover core from being broken by those modules over which the Drupal developers have less control.

The problem with tests being stuffed in a testing framework is that they tend to be forgotten... and enter a sort of "as long as they're green it's OK" state. Sometimes effort is being put in fixing tests that aren't even needed anymore, but they're so old that no-one really knows what they do and they're showing red so... fix them :)

February 19, 2008 - 12:30
Itkovian (not verified):

I think that once you have a test framework in place, which is used for core regression tests, you can easily expand it to include modules. I really think testing is mandatory, especially for a project that is powering a lot of high-traffic and high-visibility websites. It may not be easy, it may not be exciting, but it keeps the platform stable and helps uncover bugs early. I really do not grok why you would think this requirement to be a bad idea. Furthermore, breaking backwards compatibility is not a bad thing if it improves stability, performance, ease of use, etc. Would you rather like to see Drupal go the x86 way?

March 15, 2008 - 22:36
Tj Holowaychuk (not verified):

For JavaScript I would strongly recommend JSpec http://visionmedia.github.com/jspec, it is VERY light (one script), 45+ matchers at the moment, extremely readable, DOM / framework independent, supports DOM sandboxes, async requests and more.

April 8, 2009 - 08:06

Add new comment

© 1999-2014 Dries Buytaert Creative Commons Attribution-NonCommercial-ShareAlike 3.0 License.
Drupal is a Registered Trademark of Dries Buytaert.