You are here
Two weeks ago at DrupalCon San Francisco I gave my traditional state of Drupal presentation. A total of 6000 people watched my keynote live; 3000 were present at DrupalCon, and another 3000 watched the live video stream. Nonetheless, a lot of people asked me for my slides. So in good tradition, you can download a copy of my slides (PDF, 48 MB) or you can watch a video recording of my keynote on archive.org.
It's that time again. Time to look back at 2009, and to look forward to 2010.
In my 2009 predictions for Drupal, I was pretty much spot on -- except for the Drupal 7 release date. I predicted that the two most exciting features in Drupal 7 core would be custom content types and radical improvements in usability, that a number of important contributed modules would move into core, and that core would embrace the semantic web. I hoped that our community remained strong, and it did. Our community is our biggest asset and Drupal 7 wll be our best release yet, and I'm very proud of both. We accomplished much by working together in 2009, and I'm very confident of what we can -- and will -- do in 2010.
On a personal level, 2009 was a very busy Drupal year. I posted 3,269 comments on drupal.org (up from 1758 comments in 2008), and committed 1,567 patches (up from 1,031 patches in 2008). I accepted many more speaking engagements that evangelize our work, wrote more blog posts (215 blog post in 2009, up from 183 in 2008), and more.
My two personal Drupal highlights for 2009 include Whitehouse.gov switching to Drupal and the automated testing that we deployed on drupal.org. In fact, having now experienced its benefits, I'm not sure how we ever developed without automated testing in the past. The current Drupal 7 development snapshot feels more stable than the initial Drupal 6.0 release (minus obvious exceptions like no working upgrade path). Automated testing improved our development velocity as we committed many more core patches than any previous year. Other highlights include the many Drupal books that were published, and of course, the first Drupal tattoo -- both strong proof that Drupal is here to stay (or, at least, that tattoo is ;-)).
© <a href="http://www.robcottingham.ca/cartoon/2009/12/27/drupal-white-house">Rob Cottingham</a>
My personal low for Drupal in 2009 is that we didn't get the Drupal.org redesign implemented -- fortunately, we have that back on the fast lane.
2009 was also the year that Drupal started to get noticed by CIOs as illustrated by the fact that Gartner put Drupal in the visionaries quadrant and the number of Fortune 500 companies that started using Drupal. While Drupal grew in all dimensions, it probably saw the most relative growth within the enterprise. I think this is part of a bigger trend, because it feels like Open Source became almost generally accepted in 2009. Many more businesses realized that Open Source is a viable alternative, and as a result, I don't recall many Open Source "battles" in 2009 like those of the past. It is a little sad because I enjoyed fighting the good fight, and because it provided a healthy competitive edge. Reality is that, for the most part, we have won the Open Source battle. Open Source matters more every day and is changing the software industry -- already Microsoft is working on an Open Source blogging platform. These industry changes will also likely reflect, and even change, the Drupal ecosystem. In the years to come, expect competing software vendors to adopt our Open Source techniques and licenses, and expect large consulting organizations to have their own Drupal teams.
While we didn't see a major Drupal release in 2009, it was a great year nonetheless. We've used 2009 to position ourselves for continued success in 2010 and beyond. In 2010, I predict a number of stars will align: (i) the release of Drupal 7, (ii) the launch of the new Drupal.org redesign, (iii) at least a dozen of specialized Drupal distributions gaining momentum, (iv) test-driven development for contributed modules and (v) Acquia's Drupal Gardens and Buzzr (but only if they have a free tier) that will positively impact Drupal adoption. All of these initiatives are in different stages of development, but I believe there's a big chance that their combination will translate into breakout growth by the end of 2010. It also implies that 2010 will see some big changes, which is never easy.
Going forward, it is important we keep up with the larger market, which is evolving faster and faster with dozens of cool new services and APIs being launched almost every day. As a large Open Source community, we are better positioned to keep up than any proprietary vendor. We have many more people contributing to Drupal than proprietary vendors would ever be able to hire. However, proprietary vendors excel at focus and execution. There is nothing we can't do, but it is important that we're focused on the right things, and that we continue to be execution-driven. Let's remember our oldest mantra: "Talk is silver, code is gold".
As people start to build more products on top of Drupal, it is important that Drupal doesn't get in the way, and that it provides the flexibility and ease of customization that Drupal site builders demand. The winning platform will be the easiest platform to build on, not necessarily the platform that has the most flexibility. Over-generalization hurts both discoverability and adoption. Drupal's power vs. flexibility vs. ease of use is a tough balance to manage, but in general, our success in this balance has created Drupal's success. Our ability to maintain that balance is key, however, I hope and predict that for Drupal 8, we'll be very focused on improving the developer experience and lowering the barriers to participation (while maintaining Drupal's power and flexibility).
In general, I think content management systems have matured to a point where, for most people, they have relatively few differentiators at their core. That is why user experience is becoming perhaps the most important differentiator for non-niche users. The Drupal 7 usability project was a bold move but I hope we learned that bold moves aren't all that disruptive as they sometimes appear at the beginning. As Benjamin Disraeli, former British Prime Minister, put it: "Action may not always bring happiness, but there is no happiness without action.". I hope that, like 2009, we all continue to focus on user experience in 2010 because the single biggest barrier to the success of Drupal will continue to be ease of use as an end-user tool.
Mentally, it feels like we've surpassed Joomla! already, but it will take a number of years for that to trickle down and sink in, and even longer for the numbers to start reflecting that. I don't think we'll surpass Wordpress any time soon, but I do think Wordpress will continue to approach us from the bottom up. But, as Open Source goes mainstream, it won't matter all that much. In 2010, we'll start to compete against proprietary vendors, some of which will start to adopt Open Source strategies themselves. If they succeed, it could change the game because they'll bring focus and execution on top of the Open Source value proposition. Whatever happens, we'll benefit from the extra competition in 2010.
Thanks for 2009! It's been an honor to be a part of the Drupal project, and it remains so today. We have plenty of work to do in 2010 so let's all focus on making change happen.
Yesterday, an anti-Drupal ad by Microsoft was spotted in the wild; see the image on the right. The news spread on Twitter like wildfire. I said this was "interesting", not because Microsoft isn't allowed to compete with Drupal but because Microsoft is also promoting Drupal. In fact, I was flattered by the idea that Microsoft considered Drupal worthy of competition. However, it left many of us confused about the fact that Microsoft decided to both partner with Drupal and compete against it.
For me, the interesting part is not whether Microsoft is allowed to compete or not -- of course they are allowed to compete. What is interesting to me is the way Microsoft reacted. Within hours, Microsoft had noticed the small Twitter-tsunami, picked up the phone to talk about it, pulled down the ad and publicly apologized for confusing the Drupal community.
This means a lot. It is hard proof that social media like Twitter works, and that Microsoft can be great at listening and responding. It is proof that the web has changed to be more humanized, and that Microsoft understands how to build relationships online. By being transparent and human, and by publicly apologizing, they built some trust with the Drupal community (as reflected in the comments of Mark Brown's blog post), and that might actually influence people's experience with Microsoft. Plus it looks like, at least for now, Microsoft decided to promote Drupal rather than compete with it.
At the end of the day, this was the act of one Microsoft employee in India who was out of sync with the rest of Microsoft. This inevitably happens in big companies. All is good now.
The Drupal 7 code freeze triggered a lot of retrospective about Drupal core, and the Drupal core development model. Some of these discussions are collectively referred to as "smallcore".
While I'd prefer to postpone such a retrospective until after Drupal 7 is released and we're about to open up the Drupal 8 development branch, the discussion is already beginning on the web, and it is my responsibility to internalize people's thoughts and recommendations.
I've brainstormed about this a lot the last couple of weeks, and talked about it with various people. In this post, I propose 8 steps to help improve Drupal core development going forward. Some of these steps make Drupal better for end-users, some of these steps make Drupal better for developers, and other steps focus on making Drupal a better product. They might not address every concern that has been raised, but I hope they make for a good start and that they provide some focus.
We'll obviously have more thinking and brainstorming to do about Drupal 8. I'd like to thank everyone for their suggestions, for bearing with some of the growing pains, and for helping us raise our game. It is what makes Drupal so great.
The 8 steps are as follows:
- Create better separation between framework and product
- Grow Drupal by making it a better product
- Select two strategic co-maintainers
- Distributions could help, if we do them right
- Experiment with distributed revision control systems
- Backport small changes to stable releases
- Continue to streamline the patch review process
- Aim for a shorter release cycle
Step 1: Create better separation between framework and product
As people start to build more products on top of Drupal, it is important that Drupal doesn't get in the way, and that it provides the flexibility and ease of customization for Drupal distribution builders. Drupal has always been king of flexibility, and missing or poorly designed APIs have always been considered and treated as bugs.
The current state of affairs is that, while Drupal is super-flexible, we do have some hard-coded assumptions, and not everything can be reused or replaced. Clearly the work on improving our APIs needs to continue, and creating better separation between the framework and the product will continue to be one of our goals during the Drupal 8 release cycle.
Over-engineering is a real threat though. When people fall in love with framework design, they tend to decouple everything, abstracting it up, down, and sideways. Before you know it, you end up with abstractions that make Drupal harder to develop for, and that could make Drupal a lot slower. When looking for a Drupal 8 co-maintainer, I'll look for someone who can help us walk this fine line. More on co-maintainers below.
I also don't like the term "smallcore" as a term for this work. First, the term seems to drive a wedge in the community -- which I don't support. Second, the term is misleading and deceptive as core is more likely to get bigger as we improve our APIs, add more subsystems and potentially add more features. Third, the term alarms people who see the value in having a strong Drupal product, as it sounds antithetical to that goal. Let's stop saying "smallcore".
Step 2: Grow Drupal by making it a better product
Drupal 7 became more mature as a framework but it is also shipping with new features, as well as a lot of usability improvements that help make Drupal easier to use for content editors and site builders. A number of people are wondering how to undo some of the usability changes, and now dream about an "easy to undo" CMS. Making the framework better is a great goal, but not at the expense of our users.
The single biggest barrier to the success of Drupal is its ease of use. This is repeatedly shown in studies, competitive comparisons and blog posts. The broader market is impeded by Drupal's usability, and not with the fact that Drupal isn't a good framework. It is a fact. This is why usability was the primary focus of Drupal 7, why I decided to hire Mark Boulton, and why this is worth making some trade-offs for.
To focus only on the power and flexibility of the framework mainly serves to keep the Drupal insiders happy, while the market and customer base passes them by to adopt tools that do not need their specialized skills because other products solve the market's needs out of the box.
There is a long history of how successful software projects evolve. In the end, there is only ever one winner in a market segment, and typically one runner-up. Everyone else becomes irrelevant in the big picture. This is true for desktop systems (Microsoft/Apple), server systems (Unix-Linux/Microsoft), ERP systems (SAP/Oracle), databases (Oracle/Microsoft) and so on ... It is what will happen in the CMS market too. My goal as the project lead, is to lead Drupal to become the dominant web platform. We do not want Drupal to be irrelevant.
We should all agree that at the end of the day, success should be measured by the number of people working with Drupal, not by the number of people working on Drupal.
A lot of the motivation for current discussions comes from frustrations with the core development process. I would like to shift the discussion, reframe the conversation, and focus on our goal to make Drupal the number one web platform. Fortunately, creating a better separation between the framework and the product aligns with that goal, but just focusing on making the framework better won't make us succeed. We should focus on making Drupal core a better and easier to use product as that is the number one barrier for Drupal's wider adoption.
We cannot afford to let Drupal remain a niche platform. Any user or organization that depends on Drupal also depends on Drupal's ability to grow. Failing to grow and improve will mean that the easier-to-use CMS competition will pass Drupal by as those products mature. This is what I called the The Ockham's Razor Principle of Content Management Systems back in 2006 and why I'm obsessed with driving both innovation and usability -- even if they sometimes seem to be at odds. I feel very strong about that vision, and can only conclude that so far, it has worked -- we've come a really long way since 2006.
If we want Drupal to be relevant longer term, it needs to be a capable, robust product that people can use out of the box. Distributions can exist to meet different market segments, but they need to build on the usability patterns set by Drupal core, as that is how we lower the total cost and risk of adoption of Drupal solutions.
Step 3: Select two strategic co-maintainers
Every Drupal release has had one or more targets for improvement. Drupal 5 focused on the installer, Drupal 6 focused on internationalization, and Drupal 7 focused on usability. Why? Neil Drumm (Drupal 5 co-maintainer) cared deeply about the installer, Gábor Hojtsy (Drupal 6 co-maintainer) was and still is passionate about internationalization, and Angie "webchick" Byron (Drupal 7 co-maintainer) cares deeply about usability. This isn't a coincidence -- I picked them with strategic objectives in mind.
I think Drupal 8 needs to have two general areas of focus, as reflected by "Step 1: Create better separation between framework and product ". To this end I am considering picking a Core framework co-maintainer and a Core product co-maintainer. The role of the Framework co-maintainer would be to tend to APIs and base level tools, and help steer us to our goal of achieving better separation between the framework and the product. The role of the Product co-maintainer would be to make Drupal the best CMS in the world, by increasing usability and adding and improving features. My role in this plan is to assist and manage both co-maintainers, and to help with decision making, vision and product management.
By picking and empowering two people, and assigning them separate areas of responsibility, all the while coordinating with both of them to guarantee a unified product at the end of the day, I hope to increase the velocity of development for the Drupal 8 release cycle. I also intend to accomplish the goal of making Drupal easier to use, more feature rich, and at the same time more architecturally flexible.
When looking for Drupal 8 Framework and Product co-maintainers, I'll look for people who share that same vision -- but who challenge me at the same time. A number of people come to mind, but is too early to share my current thinking. Watching people during the code freeze is a very important part of the recruiting process. After all, managing bug fixes, saying "no", and stabilizing the code base will end up being a critical part of any co-maintainer's job, especially after the development phase when the branch goes into maintenance mode.
Step 4: Distributions could help, if we do them right
So how do we improve Drupal as a product? We've had Drupal distributions since the Drupal 4.6 era and I first wrote about the potential of Drupal distributions in 2005. Drupal distributions have great potential -- turnkey solutions help us compete in new and different markets, something that could help Drupal become a significant player. The number of verticals is nearly unlimited and the opportunities are numerous.
There is a risk involved with distributions as well, which means that we need to approach them the right way. The risk is fragmentation, and it is why I feel it is important that distributions build on the usability patterns set by Drupal core. Distributions will be most helpful if they are tools that give you a head start towards building a particular type of Drupal site. Distributions that steal focus away from Drupal, or become hard to administer and extend using Drupal core usability patterns, will lead to fragmentation.
Drupal has a lot of momentum today, but we are still a niche player in the bigger CMS market. Drupal feels bigs, but we're still small. If by enabling more distributions all we do is create a lot of inconsistent niche products, some of which will be competing with one another, the Drupal project will lose momentum. If we can build competitive distributions that strengthen and accelerate the shared Drupal brand by using shared design patterns and user experience, we'll win.
Another risk is that small incompatibilities among distributions can drive distributions further and further apart. When distributions start building their own communities of contributors, having their own issue queues, all disconnected from drupal.org, then we have a problem. The difference between a distribution and a fork can be subtle. A proliferation of incompatible, incomplete and disconnected distributions could quickly become problematic. It would make it a lot harder for all of us to support and market Drupal.
We have written a lot of the the packaging scripts needed to put distributions together for Drupal.org, but we haven't decided yet how we want to use it and what a winning strategy could look like. What remains is a strategy discussion, not a technical discussion.
As a community, we have to take responsibility, and make sure that distributions collaborate rather than compete, much like the way that we attempt to work together on modules. That starts by centralizing all the code on drupal.org, by making Drupal core flexible enough, but also by encouraging shared design patterns and user experience. With distributions, community responsibility and leadership becomes even more important. Building one product is hard. Building a set of products in a way that strengthens and accelerates Drupal is much, much harder.
Step 5: Experiment with distributed revision control systems
If the road to success is making Drupal a better product (along with making it a better framework), we need to be able to add new features to Drupal core.
Some features in Drupal core, like forums, blogs, polls and aggregator, haven't evolved as fast as the Drupal framework itself. For many people these modules are fine as they are, but more advanced alternatives have emerged in the contributed module repository. It begs the question: should we advance the modules in core or are they sufficient for 80% of our users? But also; are there any modules that we should be adding to core to make Drupal a better product?
I think we should advance those modules, and add some more. This is what I want the Product co-maintainer to help me with.
Giving more CVS write accounts or splitting core in smaller projects could help those modules advance faster, but it will also hurt us. A much better implementation to accomplish the same goal seems to be to experiment with distributed revision control systems. There are various technical issues to sort out to make this feasible, but I think it is worth the experiment because it will allow us to interact in new ways; ways that empower individual contributors, ways that allow people to self-organize around features, and that enable me to pull in bigger changes in a controlled fashion so we maintain consistency, quality and vision. To do it in a way that I'm comfortable with, it will require a significant investment in the project module and the Drupal.org infrastructure though.
In other words, I see myself experimenting with a distributed revision control system like Git or bzr. Not immediately, but somewhere in the Drupal 8 development cycle. Right now, I want everyone to be laser focused on getting Drupal 7 out. At least initially, I'd want the CVS repository to be the main, canonical repository from which Drupal core is packaged. However, I'd additionally pull in changes through Git/bzr for some period of time, to help decide if Git/bzr is what we want.
Step 6: Backport small changes to stable releases
Some people suggested that the framework and the product could move at different speeds. I am uncomfortable with that concept.
Feature development drives API advancements. API improvements drive new and better features. It's an important feedback loop. My conviction is that feature development and API development are so related that it would be detrimental to try to do them asynchronously. Drupal's APIs have been changing for 10 years and there is no reason to believe that they will all be stable one day.
That said, there might be certain changes, like smaller usability improvements, standalone features, or new hooks that don't break existing code, that could be backported to previous Drupal releases during the development cycle. Of course the changes cannot break any code or invalidate existing documentation, but I think having a good test harness will help. Going forward, I'd be comfortable backporting certain changes that help Drupal advance, as long it is transparent to existing Drupal installations.
Step 7: Continue to streamline the patch review process
A key problem still seems to be that it is difficult to get solid patch reviews, making it slow for some patches to get committed. Finding solid reviewers is hard, especially as Drupal becomes more advanced. I still believe this is a primary source of frustration, and that we should continue to streamline the patch review process.
We've introduced new rules in the patch review process lately (e.g. wrapping at 80 character length, formatting of PHPdoc, capitalization rules, etc). The proliferation of rules makes it more difficult to get your patch in. As a frequent reviewer and committer, I've found that code-style reviews, while important, can add quite some noise to an issue, and that it can derail API and architecture review.
One way to streamline the patch review process is to automate code style reviews just like we automated testing -- at least to the extend possible because coding standards can be both complex and subtle. In an ideal world, code style reviews would take almost no comments, and would be reduced to a green or red status message just like with our automated testing. It could reduce the size of the 'needs review' queue further to things which actually merit review etc. By doing so, it would provide more focus, and help improve our velocity.
It doesn't necessarily solve the problem that one can't find enough good reviewers for one's patch but I think it will help.
Step 8: Aim for a shorter release cycle
There is a sentiment amongst developers that the Drupal 7 core release cycles was too long. It can make it frustrating to contribute to Drupal core development. Who wants to work on a feature that might not be used in production for another 18 to 24 months? Near the end of the code freeze, things heat up, and people submit a lot more patches. In other words, shorter release cycles could make it more compelling to contribute.
At the same time, I don't think the Drupal 7 release cycle was too long, per se. A lot of the big new features, such as the new database abstraction layer and Field API (formerly known as Content Construction Kit (CCK)), took several months of work, and many months more to be usable. We finished converting code to the new database abstraction layer over a year after the initial commit. Drupal 7's Field API is probably the biggest architectural change in the last 5 years -- it will take a couple more major releases for Field API to be truly integrated into Drupal. In other words, sometimes we need a longer release cycle to allow for big, game-changing changes.
How long the Drupal 8 release cycle will be I can't say yet, but let's aim for a shorter release cycle. The length of the release cycle depends on how fast Drupal 7 is adopted, how fast the contributed modules reach the "plateau of productivity", and how fast Drupal 8 shapes up to be a great release. You don't want it to be too long, but you don't want it to be too short, either. Furthermore, as a goal, the length of the release cycle shouldn't trump the larger, overarching goals of any given release. I'll take constant temperature of where we are and balance all the different pros and cons (eg. innovation versus stability).
Another adjustment to the release cycle could be separate code freezes for the Framework (APIs) and the Product (user features and usability). The Framework/API freeze would precede the Product/CMS freeze, and would let us focus on establishing the underlying changes earlier in the overall cycle, with focus shifting to features and usability later in the cycle. This will make feature development easier as the APIs will not be shifting around as much, and it will give a period of time after the Framework freeze where we can focus on bugfixing and performance improving the Framework, even while new features are still being developed.
Some closing thoughts
The 8 steps outlined above describe actions and steps that we can take to help make Drupal 8 rock. As we look forward to Drupal 8 development, I think it is important to focus on the right things. If we want Drupal to win and prevent Drupal from being stuck as a small player in the bigger picture, we need to focus on making Drupal easier to use for both end-users and developers, while maintaining our innovative edge. Some of these 8 steps make Drupal better for end-users, some of these steps make Drupal better for developers and some of these steps make Drupal a better product. I truly believe we can all win.
In my DrupalCon Paris presentation I talked about what it means for Drupal to grow up -- and I wanted to elaborate on that a bit more in this blog post. I hope that the analogy that I'll use in this post can provide a framework for thinking and discussion about it.
We, as a community, are growing up and there is not much you can do about it.
If you're my age (currently 30 years old), sometimes you remember how great it was when you were in your teens. Unfortunately, you have no choice to grow up: you can't roll back time nor freeze it. I think this will ring true with most of us, and frankly, the same is true for Drupal: Drupal is growing up, and we have no choice but to grow along with it. Growing up is inevitable. Life changes every day and excessive nostalgia kills happiness.
So if Drupal is growing up, where is it in its life?
For me, Drupal is a young adult in the early phases of its professional career. Drupal is fresh out of college with A-grades, did some highly-visible internships while in college, landed its first job in a high-profile company, and built up some initial work experience. He has everything it takes to become successful, but being a junior team member, hasn't yet proven himself in a big way. He has the raw talent to become a key part of the business. In fact, his first promotion is just weeks away, and it remains to be seen how he'll handle some additional responsibilities. Either he is happy with his life as it is, and takes it the easy way, or, instead, embarks on a bigger career path in a somewhat naive but admirable desire to conquer the world.
But, enough with the analogies. For Drupal, growing means we must continue to innovate at the framework layer by improving our code, our tools and our developer workflows. We have to continue to do what we have been doing the best. But, there is also a really big "and" that is key to us growing up ...
As a community, we have to embrace increasingly more end-users, content editors, designers, usability experts and organizations. It may sound obvious, but we have to learn to build software for the people that are our users, rather than mainly designing for ourselves like we've always historically been doing. We, developers, should be the primary target audience of "Drupal: the developer framework" and we should continue to invest heavily in it. But end-users, and content editors in particular, have to be the primary audience of "Drupal: the content management system". Both areas have to thrive and work together. We can either succeed at making that happen with a somewhat naive but admirable desire to conquer the world, or we can fail at making that happen and remain insignificant in the bigger picture.
There is a lot of richness in the Drupal platform that we haven't really figured out how to package in order to reach many more people. Drupal 7 will hopefully be a big help with that, but we'll need to continue that trend with Drupal 8 and beyond. Doing so may provide some initial discomfort as we break out of our traditional mindsets, but it is also tremendously exciting. It's like getting a promotion.
At the end of the day, it is all part of growing up and part of Drupal's natural evolution as a product and technology. Growing up is inevitable -- you can't freeze time. Part of growing is learning to take on more and bigger problems. It is no coincidence that the biggest challenges tend to be ahead of you. This is true for your personal life as well as for the life of an Open Source project. Being a young adult is one of the most exciting times of life, and is filled with lots of changes. What's not to like?
Maybe in a few year's time, I'll write about how Drupal is getting married, and that they are talking about getting kids. ;-)