Dries Buytaert

In your keynote at DrupalCon Boston, you said you were not sure that Simpletest was not the right Framework. What are the other options you are considering, or is a custom framework desirable?

The main reason I am asking is that some people see creating tests as a meaningful way to contribute to the Drupal project, but we don't want to make them in SimpleTest, if it is not the long term choice. Thanks.

"Xunit as a framework", meaning, use a variety of unit testing tools to test.

In java, this is a common set of frameworks
JUnit
DBUnit
XMLUnit
WebUnit
Watir (firefox, IE web browser automation)

And transcending utilities
EasyMock

A common occurance is unit tests originally written to framework changed to use another framework or split into 2 tests, as a technology was introduced into a module. For example changing a test written using JUnit might have been ported to use XMLUnit as XML was introduced into a module. It's a natural progression. Or having tests at multiple levels
UserTest
UserDbTest
UserUiTest
Is normal too.

So responsibilty fell on the more senior folks to look at when tests became to cumbersome to write, maintain or debug because they dealt more with setup and technology and not with actually testing the business logic. Convienience methods (plethora of methods on DrupalTestCase) are not a solution, but a stop-gap measure.

I mentioned this to folks the idea of using a "best of breed" set of frameworks at Drupalcon and response was that it was fragment and confuse folks. This is a valid concern, "competeing" frameworks like PHPUnit and Simpletest would cause confusion. Simpletest is a decent lowest comment denominator and if some folks never leave there, that's ok. It's important not to shackle folks from leveraging useful frameworks.

I realize PHP doesn't have as many "XUnit" packages, but there may be more than you think. On that note, "This is DBUnit for PHP" sometimes doesn't always translate well. In particular, static typed languages like java lead to heavier frameworks and do not leverage the dynamic nature of scripting languages, but I digress.

We've a lot to learn about testing and test frameworks so we're likely going to want to make some changes down the road. At the same time, it is important that we pick the right tools so we don't waste people's time and code. If can happen, but we should try to avoid it.

As a first step, I'd like to invite you to take a closer look at the SimpleTest framework for Drupal and to share your thoughts. But maybe there are some other frameworks out there that we can learn from, or get inspired by. Let's collect some recommendations.

I'm inclined to say that we should strip down SimpleTest and ship a minimalistic version of it with Drupal core. There are also some aspects of SimpleTest that are somewhat lacking such as its sandboxing functionality (it doesn't create a test database or a test directory for uploading files), the fact that it is difficult to run from the command line, etc.

I'm still learning SimpleTest myself and I'm not too worried about its shortcomings. In good Drupal tradition, we can incrementally improve the test framework over time. What is important is that we have a powerful yet lightweight base to that sets us of in the right direction.

I plan to play around with SimpleTest some more next week, but I'd recommend that someone creates a critical issue on drupal.org and actually starts working on a stripped down version of SimpleTest. By moving this to the drupal.org issue queue, we're more likely to make progress.

For a live status report of the unit testing progress, have a look at the Functional SimpleTest Status page. It lists a bunch of tasks and issues, as well as how they're doing.

I've been using Simpletest for a while and I think it's a reasonable good option. However I don't think we should just use Simpletest and then make Drupal development 100% dependant on it.

What I mean is that we need to define our own Drupal Test API (Like making all the test classes extending DrupalTestCase) so if at some point it's not working for us anymore we can just drop in a different test framework.

Also it may be good to have separated tests for pure Drupal API tests and for functional tests (the ones needing page loading and a browser like engine), which will allow us to use other tools for that ones, Maybe in the future we can even use some js capable testing engine ...

So, short summary, ok for simple test just but we better use our own testing api (and this will allow us to just start writing tests now, not waiting months for the perfect testing framework to be there..)

The push to bring automated testing to Drupal is definitely exciting. After working on a couple of projects that took different routes to that goal, I think there are some important rules to remember that there's a big difference between a 'test script simulator' -- something that, say, uses the UI to create a node and publish it -- and API level unit tests. Both are needed if we're to have anything approaching complete coverage.

The 'test script' style unit tests are best at capturing the steps to reproduce a given defect, to ensure it doesn't reappear (I.E., the infamous unchechable checkbox issues with FAPI). The API-level unit tests are what can be used to ensure that things continue working the way we expect throughout Drupal's codebase when a change is made.

Most of the tests I see being written write now are the 'automated test script' variety, and they're testing positives ('can I create a node?') rather than negatives ('when I create a node with X options, using Y steps, does defect Z show up?'). These are useful as smoke tests for general functionality, but they'll really give us a false sense of security if we treat them as the be-all end-all.