Dries Buytaert

The future is a RESTful Drupal

Last weekend, we held a sprint at the Acquia offices for the Web Services and Context Core (WSCCI) Initiative for Drupal 8. This was an important sprint for the future of Drupal. This blog post provides a high-level overview of what was discussed and agreed upon; the different sprint participants will be laying out more technical details in follow-up blog posts.

Overall, a wide range of experience levels, skill sets, and perspectives were brought to the table, with the goal of the sprint being to clearly define the initiative’s scope, get agreement on what we wanted to accomplish and why, and lay out a clear plan for how to accomplish this.

In attendance were:

• Larry "Crell" Garfield, lead architect of the WSCCI initiative
• Daniel "sun" Kudwien, top Drupal core contributor
• Fabien "fabpot" Potencier, lead developer of the Symfony project
• Kris "EclipseGc" Vanderwater and James "neclimdul" Gilliland, who have helped develop and implement the plugin system as part of WSCCI
• Greg "heyrocker" Dunlap, initiative lead of the Configuration Management Initiative for Drupal 8
• Moshe Weitzman, long-time Drupal core contributor
• Angela "webchick" Byron, community cat herder and Drupal 7 core co-maintainer
• Randy "rfay" Fay, Justin "beejeebus" Randall and Alex "effulgentsia" Bronstein, core developers and sane voices of reason with an outside perspective
• Dries Buytaert (me), project lead

Scope

The WSCCI initiative, as envisioned by Larry Garfield, was originally set to address Drupal's web services and flexible page layout capabilities. We discovered that both would require significant changes to Drupal core, and it was difficult to build consensus online, so we decided to get together for 3 days and to flesh out what we actually wanted to accomplish, and how.

At the sprint, we first attempted to articulate all of the problems that WSCCI was trying to solve, which included: multiple page layouts, better UI/UX to manage blocks, partial page rendering (ESI, AHAH), contextual blocks, different response types per delivery callback/URL, plugin system / swappable subsystems, lazy loading bootstrap (convert subsystems to PSR-0 classes), infrastructure for building web services, dependency injection, and so on.

We then did a round of voting where we could each choose 3 of those things in order to try to determine which of those were the most important.

Two things became instantly clear during this exercise:

1. The items encompassed under WSCCI really spanned at least 3 separate major areas: Web Services, more robust ESI-based layouts (think Panels only more powerful), and cleaning up our underlying toolset to be a more loosely-coupled framework.
2. The underlying architecture to support RESTful calls to Drupal that makes all of the other things possible was deemed the most important thing to focus on.

Scope resolution

After a good chunk of discussions, all were in agreement to scale back the scope of the initiative to just the "Web Services" piece, and spin off the Layout/blocks/related-UI parts to a separate effort.

Furthermore, some efforts, such as PSR-0 and Unified Plugin system, were only semi-related to the WSCCI initiative in the first place, and just happened to become relevant for it. Work on those efforts will continue as part of the general Framework community efforts.

Architecture

Fabien was able to offer a tremendous number of insights as to how various Symfony2 components could help provide underlying structure for Drupal core to support Web Services out of the box. Essentially, most of what the WSCCI team had been trying to work toward, in the abstract, was already implemented within Symfony2. While some implementation details were different than what we had in mind, the end result is almost exactly what we have been trying to accomplish. We therefore agreed that the best way forward was to leverage several Symfony2 components within Drupal as a way to speed progress toward that end.

Benefits

Some of the concrete benefits that would come out of these changes are

• An underlying framework that is a first-class REST citizen. That means developing web services becomes easier and more efficient, because the plumbing is already in place. An HTML page is a particular case of a REST service.
• Because the core system will be fully REST-ified, we'll be able to improve existing APIs and expose Drupal content in a natively RESTful way. For example, our current AJAX system doesn't comply with REST standards, but with these changes, can be cleaned up to do so.
• That in turn makes Drupal-to-Drupal communication, content staging, content sharing, and a host of other related tasks easier.
• The use of widely used libraries and techniques makes Drupal more approachable to new developers.

Why does this matter?

As it has evolved into an increasingly powerful system, Drupal has gotten increasingly complex and is not as easy to start developing with as it once was. Many developers are nervous about continuing that trend. Managing complexity is a challenge faced by many software projects, and one approach is to use standardized patterns and components.

Due to its long support for PHP 4, as well as some unique needs, Drupal does not take full advantage of standardized patterns and components. The complexity of the custom code that’s used and the non-standard architecture combines to create a barrier to entry for developers new to Drupal (both experienced and novice developers alike).

Meanwhile, the web is constantly changing around us. Web services and mobile are more important than ever, and with that comes the need to have more flexible page and layout capabilities. Now feels like the right time to modernize Drupal’s capabilities to bring it to the forefront of modern PHP systems and web systems in general.

While changing Drupal's core plumbing to address these needs, it's also a good opportunity to do so using more widely understood and modern techniques and libraries. Leveraging the work of a fellow open source community lets us get a jump on these changes to build a more powerful, more flexible, and more easily learnable system in less time than it would take to go it our own.

While these changes may seem large, and some of them are, we believe that they will achieve the right balance between empowering Drupal's design and architecture, and moving toward more modern, standard, well-tested code and techniques to empower a new generation of developers. I hope you are as excited as we are!

One Drupal to rule them all

Ten years ago, the average organization had one website. Since then, the world has become a more complex place with a diverse set of needs. If you're like most organizations the number of sites you have continues to grow at a rapid clip. You've dipped your toe into the social media waters by setting up one or more blogs, you use microsites for the foundation of your marketing efforts to promote products and events, and community sites to engage with the people who use your products. And of course, you have your corporate website, as well as your intranet and a number of internal collaboration websites for different projects. This is today's reality.

During the past year, I've met with many organizations that have hundreds of sites; some even have thousands of sites. Most of these sites are vastly different in terms of scale, functionality, complexity and longevity. As a result, the level of investment and the time to market requirements are usually very different. Some of the websites are owned by the company's IT department while other websites may be owned by their marketing department.

For most organizations, one tool cannot get the job done, so they keep multiple tools in their toolbox - whether they intend to or not. Unfortunately this is a common scenario in many enterprises. We see many organizations that run on Vignette, but they add WordPress to power their blog, and use SharePoint for their intranet. Managing integration and multiple (proprietary) solutions is not only costly but it acts as a roadblock to innovation and slows time to market when changes are needed.

It can be a complete mess.

CIOs are now waking up; they're realizing that the cost of running twenty different content management systems on twenty different stack configurations is an expensive, unnecessary burden for the organization. They can see that there are cost savings to be made if they standardize. What they want is something more than a content management system: they need a single platform that meets their needs. They need Drupal.

The New York Stock Exchange is launching 30 sites on Drupal. Turner Broadcasting is migrating 80 sites to Drupal. A large consumer pharmaceuticals company might end up with over 3000 Drupal sites. Why? Because Drupal gives their IT team an agile platform to deliver new sites quickly with the breadth of capabilities to meet their needs, but broad enough in scope to serve as a web platform too.

More and more, I see large organizations standardize on Drupal. Drupal's low-cost entry point and open, modular architecture provides the perfect foundation. Drupal is one of the few solutions that can scale from very small to extremely large, and has the depth and breadth of functionality to support thousands of different use cases. Drupal sites can span a wide range of functionality; from blogs, to marketing microsites, social business community sites, corporate intranets, e-commerce sites and more. There are incredible success stories for each of these use cases already. Plus, we've now reached a point where many open source content management systems outperform proprietary solutions on technical superiority and pace of innovation. Drupal is in a very unique position.

This is an immensely powerful trend. If we navigate this well, Drupal could become truly ubiquitous and go from powering 1% of the web, to quickly powering 5% of the web and even further. It requires us to keep making Drupal a better platform, to crack the code on how to make Drupal distributions truly compete with commercial point solutions, and to build out an even larger commercial ecosystem and community. Needless to say, I'm super excited about Drupal's future and the opportunity ahead of us.

Continuous services and connected devices

Ray Ozzie, who took over the role of Chief Software Architect at Microsoft when from Bill Gates retired about five years ago, announced recently that he will be retiring soon. When Ozzie first became Chief Software Architect, he wrote a famous 5000-word internal memorandum titled, The Internet Services Disruption. The memo outlined the transformative and disruptive potential of web services.

Ozzie wrote, "The ubiquity of broadband and wireless networking has changed the nature of how people interact, and they’re increasingly drawn toward the simplicity of services and service-enabled software that ‘just works’. Businesses are increasingly considering what services-based economics of scale might do to help them reduce infrastructure costs or deploy solutions as-needed and on subscription basis.". Ozzie was spot on as this is what today's Software as a Service (SaaS) and Cloud Computer are all about.

Now, five years later, just before he is about to retire, Ozzie wrote another 5000-word memo entitled Dawn of a New Day. In this memo, Ozzie reflects on how Microsoft has been transformed over the past five years with regards to so-called 'services'. Despite many successes, Ozzie acknowledges that for the most part, Microsoft missed the boat on mobile and social software.

More importantly, he also lays out his vision of where things are going in a post-PC world: "To cope with the inherent complexity of a world of devices, a world of websites, and a world of apps and personal data that is spread across myriad devices and websites, a simple conceptual model is taking shape that brings it all together. We’re moving toward a world of 1) cloud-based continuous services that connect us all and do our bidding, and 2) appliance-like connected devices enabling us to interact with those cloud-based services."

I spent most of my evening reading both memos. It provides some unique insight about what it means to be Chief Software Architect at one of the largest software companies in the world, as well as how they see the future. In general, I agree with Ozzie's vision of the future as he explains it in his latest memo. The part on complexity also resonated with me. I think the points he makes are very relevant for most of us that make a living with Drupal. Like Ozzie, I think development of this new service-connected world will be neither fast nor easy. However, I think Drupal is uniquely qualified to play a prominent role in such a world. It requires us to make the right decisions, to manage complexity, and to stay on top of our game. Whether you like Microsoft or not, the memo is worth reading.

Drupal in a tablet world

A few years ago, computer tablets similar to Apple's iPad were props in science fiction films. Only a couple of years from now, tablets might be among the most popular consumer electronics ever.

It took less than three months to sell the first 3 million iPads. This has made the iPad the consumer electronics device with the fastest adoption rate of all time. Compare that with the 1 million iPhones sold in the first three months of its release, and the 350,000 DVD players sold in the first year of their mass production.

According to CNBC, the iPad will surpass gaming equipment and the cellular phone to become the fourth most popular consumer electronics category -- televisions, smart phones, and notebook computers are the top three. Large companies like Hewlett Packard have no choice but to design credible alternatives to the iPad. Samsung will soon begin to distribute their Galaxy Tab, which runs on Android OS, in Europe and the U.S. Once these big players enter the market fully, it's pretty clear that we're heading to a new lifestyle in which everyone and his dog owns a tablet. I already have mine.

This all begs the question: in a world of tablets and mobile handheld devices, what do we need to do so that Drupal will be a go-to platform? How can Drupal contribute so as to be a player in the ever-expanding ecosystem of tablets and mobile phones? What needs to change so that developers can build easily tablet and mobile phone optimized applications on top of Drupal? At the rate that tablets and mobile devices are being adopted, we don't have much time to get in motion. Drupal 7 takes enormous steps in the right direction, and I have a some ideas for Drupal 8. However, I'd like to get your take first ...

Thoughts?

Drupal 7, the cocoon and the butterfly

There exists an interesting story about a man and a butterfly cocoon. It is about a man that found a cocoon, and brought it home to watch it turn into a butterfly. As the butterfly inside matured, it struggled to get out of its cocoon, but couldn't quite get free of it. One day, the man became tired of waiting and decided to help the butterfly. He removed the remaining bit of the cocoon. The butterfly was pleased, but it had a swollen body and small, wrinkled wings. As a result, the butterfly never succeeded in flying and spent its entire life crawling around.

What the man didn't understand was that the struggle required for the butterfly to break out of its cocoon actually forced fluid from the body of the butterfly into its wings so that it would be ready for flight once it achieved its freedom. It is the struggle that causes the butterfly to develop its ability to fly.

I feel the same way about Drupal 7. Seeing Drupal 7 getting steadily closer to its release, is like watching a cocoon grow into a butterfly: the inevitable results are going to be spectacular. Release management and fixing bugs is hard work, the work of a determined caterpillar. However, I think Drupal 7 will be quite a metamorphosis relative to Drupal 6. Not only will it look different, it will function differently -- making users and developers feel like Drupal spouted wings.

And like the caterpillar that grew into a butterfly, it won't be the Drupal many of you used to know. It will be a nicer looking, more powerful, and more scalable Drupal that is easier to use. If you overlooked Drupal previously in favor of another system, you might want to revisit Drupal once Drupal 7 is released. I think you will be surprised at the difference. Or if you know someone that overlooked Drupal in the past, you might want to echo this story.

What I want for my website

I really only want two things for my website: (1) I want the software that runs my website to be high-quality and (2) I want my website's content to be high-quality. It sounds easy and straight-forward but I assure you it isn't.

I want the software that runs my website to be stable, efficient at handling my website's traffic, and flexible. Good content management systems meet these requirements, but it took years to get where we are today, and we still have a really long way to go. Fortunately, all my websites run Drupal, so the first part of my requirements presents no problem. If you want, you can have a Drupal site too -- it's free! :)

I also want the website content to be of high-quality. That applies to both the quality of my own writing, as well as that of others that participate on my site. I believe this is a much more difficult problem to solve -- I'm interested in helping to solve it.

I'll be moving from an apartment to a house with a small yard, so last weekend we bought a hose to water the plants and the grass. Now that I'll have to get into gardening, it struck me that maintaining and improving the quality of my website is a bit like gardening too.

First and foremost, you have to keep the weeds out. In the world of websites, that means preventing spam comments and other unwanted posts. I never liked weeding -- as a kid, I had to help weed our garden. Collecting a bucket of weeds each vacation day was no fun. I don't like manually deleting spam comments either, so I addressed that with Mollom.

So far so good -- Drupal and Mollom make me a pretty happy camper as my two requirements are mostly met.

Just weeding your garden, though, doesn't make it beautiful or interesting. The same is true for websites. My favorite websites are those where the quality of the comments exceed the quality of the actual posts. I think there is much more that can be done to improve the quality of content on websites.

For example, I often wish I could tell people that their comments are poorly written or formatted before they even submitted it. It would be nice if more comments used proper capitalization and punctuation like we learned it in school. Or better yet, imagine having a service that estimates the added value of any new comment, or that somehow encourages thoughtfulness and constructive debate. I know I'm dreaming (and rambling a bit), but I also believe that these kind of tools are in our future for those that want them. They would be a natural addition to Mollom so maybe I'll be working on them myself, especially if people keep getting their capitalization and punctuation wrong. ;-)

On business models for Drupal distributions

The fact that thousands of developers use Drupal to make money building websites for their customers has resulted in thousands of modules being created and hundreds of events being organized around the world. When I started Drupal, I wasn't aware of the importance of such a commercial ecosystem. Looking back at 10 years of working on Drupal, it is an important lesson learned. If I were to start a new Open Source project (I'm not!), the ability to build out a large commercial ecosystem would be one of the criteria that I'd look for. Disruptive innovations change entire industries, not just tools. Not every Open Source project lends itself to that.

I'm repeating myself, but if we want Drupal to be relevant longer term, one of the things we need to do is "make Drupal distributions work". Drupal distributions allow us to compete with a wide range of turnkey solutions as well as invent new markets. The number of different distributions we could build is nearly unlimited. From what I can tell, Drupal is the only Open Source content management system that is actively encouraging its community to build and share distributions. We have a very unique opportunity in front of us -- distributions can be a game changer.

But what does it mean to make Drupal distributions work?

We've began work on Drupal distributions during the Drupal 4.6 era based on our experience with CivicSpace (a distribution for political campaigns). Drupal 5 was a big milestone as we introduced a web-based installer with support for install profiles. We made incremental improvements to install profiles in the Drupal 6 release, and it wasn't until Drupal 6 that we saw a number of great Drupal distributions emerge: OpenAtrium (an intranet distribution), Acquia Drupal (a convenience distribution for site builders), OpenPublish (a distribution for online publishers), Pressflow (a distribution with performance and scalability improvements) and more. Finally, with some of the install profile related improvements in the upcoming Drupal 7 release and the fact that we can build and host distributions on drupal.org, I expect to see many more distributions going forward. In summary, we evolved the underlying technology over the course of 5 years and might have reached a point where our vision of install profiles can really come to live.

But ...

While we made a lot of progress on making distributions feasible from a technical point of view, we have yet to figure out the business model around Drupal distributions. Building and maintaining a high-quality Drupal distribution is no small task. It is also different from contributing a module. While writing a module is often billable, maintaining a Drupal distribution is arguably less so. In other words, can we build a successful commercial ecosystem around distributions so that we'll see hundreds, if not thousands of high-quality distributions, flourish?

We need to figure out how to make it commercially interesting (or at a minimum, commercially viable) for organizations to invest the time and money it takes to build and maintain a distribution. If not, distributions risk being nothing more than a costly but fun lead generation tool. I don't think that is scalable. To make Drupal distributions the game changer it could be, it has to be a no-brainer for organizations to get into the game of building one. Reducing the maintenance cost through tools like Drush Make and the packaging infrastructure on drupal.org certainly helps, but is probably not enough to make distributions take off in a big way.

At Acquia, it occurred to us that we might be able to help. Many Drupal shops lack the go-to-market infrastructure that Acquia built out over the last 2.5 years (i.e. 24x7 help desk, a marketing and sales organization) and that products often need. We can help market and sell offerings around distributions (e.g. 24x7 SLA-based support, hosting, remote administration) and share the revenue with the organization actually building and maintaining the distribution. It is a well-known model in the software world (such as the game industry), and is one example of how we could try to make it commercially interesting to build and maintain distributions. For more information about this, I recommend reading Tom's blog post on the 'Software Publishing Model'.

Four Kitchens has built a business around offering consulting and support for Pressflow, the distribution they authored. Pressflow's popularity has driven demand for these services, creating a unique positioning and opportunity for Four Kitchens. Development Seed is in the early stages of rolling out their business model for OpenAtrium, one of the distributions they have created. They announced plans to offer developer support and a paid partner program as key tenets of their business model.

Of course, these are only a few examples of how we can help make Drupal distributions work. As a community, I think we need to brainstorm about this more.