Dries Buytaert

Drupal in the cloud

It is not always easy to scale Drupal -- not because Drupal sucks, but simply because scaling the LAMP stack (including Drupal) takes no small amount of skill. You need to buy the right hardware, install load balancers, setup MySQL servers in master-slave mode, setup static file servers, setup web servers, get PHP working with an opcode cacher, tie in a distributed memory object caching system like memcached, integrate with a content delivery network, watch security advisories for every component in your system and configure and tune the hell out of everything.

Either you can do all of the above yourself, or you outsource it to a company that knows how to do this for you. Both are non-trivial and I can count the number of truly qualified companies on one hand. Tag1 Consulting is one of the few Drupal companies that excel at this, in case you're wondering.

My experience is that MySQL takes the most skill and effort to scale. While proxy-based solutions like MySQL Proxy look promising, I don't see strong signals about it becoming fundamentally easier for mere mortals to scale MySQL.

It is not unlikely that in the future, scaling a Drupal site is done using a radically different model. Amazon EC2, Google App Engine and even Sun Caroline are examples of the hosting revolution that is ahead of us. What is interesting is how these systems already seem to evolve: Amazon EC2 allows you to launch any number of servers but you are pretty much on your own to take advantage of them. Like, you still have to pick the operating system, install and configure MySQL, Apache, PHP and Drupal. Not to mention the fact that you don't have access to a good persistent storage mechanism. No, Amazon S3 doesn't qualify, and yes, they are working to fix this by adding Elastic IP addresses and Availability Zones. Either way, Amazon doesn't make it easier to scale Drupal. Frankly, all it does is making capacity planning a bit easier ...

Then comes along Amazon SimpleDB, Google App Engine and Sun Caroline. Just like Amazon EC2/S3 they provide instant scalability, only they moved things up the stack a level. They provide a managed application environment on top of a managed hosting environment. Google App Engine provides APIs that allow you to do user management, e-mail communication, persistent storage, etc. You no longer have to worry about server management or all of the scale-out configuration. Sun Caroline seems to be positioned somewhere in the middle -- they provide APIs to provision lower level concepts such as processes, disk, network, etc.

Unfortunately for Drupal, Google App Engine is Python-only, but more importantly, a lot of the concepts and APIs don't map onto Drupal. Also, the more I dabble with tools like Hadoop (MapReduce) and CouchDB, the more excited I get, but the more it feels like everything that we do to scale the LAMP stack is suddenly wrong. I'm trying hard to think beyond the relational database model, but I can't figure out how to map Drupal onto this completely different paradigm.

So while the center of gravity may be shifting, I've decided to keep an eye on Amazon's EC2/S3 and Sun's Caroline as they are "relational database friendly". Tools like Elastra are showing a lot of promise. Elastra claims to be the world's first infinitely scalable solution for running standard relational databases in an on-demand computing cloud. If they deliver what they promise, we can instantly scale Drupal without having to embrace a different computing model and without having to do all of the heavy lifting. Specifically exciting is the fact that Elastra teamed up with EnterpriseDB to make their version of PostgreSQL virtually expand across multiple Amazon EC2 nodes. I've already reached out to Elastra, EnterpriseDB and Sun to keep tabs on what is happening.

Hopefully, companies like Elastra, EnterpriseDB, Amazon and Sun will move fast because I can't wait to see relational databases live in the cloud ...

Spam, OpenID and Mollom

There is an interesting discussion about spam and OpenID going on at Matt Mullenweg's blog. The discussion was triggered by the policy decision of social bookmarking site Magnolia to restrict signups to OpenID users. According to the site, 75% of new accounts were being created at Magnolia by spammers using automated tools (our friends the 'spambots'). They say that by restricting access to OpenID users, the rate of spam-account creation decreased. In the discussion, there is a lot of talk about whether OpenID should be used to fight spam, and whether it could be an effective spam-fighting tool in the long term.

Here are my thoughts. Spammers can create OpenIDs too, and a single sign-on system might be many a spammer's wet dream. It gives them easy access to millions of sites in one fell swoop.

Now, OpenID by itself can't prevent spam. All it does is provide a globally unique identifier for any given user on the planet. This is where a tool like Mollom comes in. At Mollom we're already maintaining an internal reputation for each OpenID account we encounter while assessing submitted content. Combine an identity system (OpenID) with a reputation system (Mollom) and it becomes a lot easier to separate spam users from non-spam users. Simon Willison said it best: "a trust system requires identity first". A globally unique identifier combined with reputation tools give us a powerful weapon to fight website spam. OpenID's attribute exchange might become Mollom's best friend ...

Similarly, Tim Berners-Lee is experimenting with combining FOAF ("friend of a friend") and OpenID to fight spam: you can only comment on Tim's blog if you are no more than a certain number of degrees of friendship away from him. Of course, it is a widely accepted theory that we are only six degrees away from everyone in the world so I do wonder how effective this would really be in the long run.

It is still early days in these debates and experiments, but for now, Mollom can already protect your login and submission forms with an image or audio CAPTCHA.

Either way, it is an interesting discussion that makes you wonder. Where will OpenID be in 3 years? Where do you think the website spam problem will be in 3 years? How will this affect online communities?

I have my own thoughts and predictions and it was one of the principal reasons for co-founding Mollom ...

State of Drupal presentation (March 2008)

Last week at DrupalCon Boston I gave my traditional state of Drupal presentation in front of 850 Drupalistas. The video of the presentation is provided below, and you can download a copy of my slides (PDF, 15 MB) as well. The video is available in alternative encoding formats from archive.org.

Topics I talked about: the Drupal 6 release, the state of our union, the need for a drupal.org redesign, the Drupal 7 killer release, the Drupal 7 development cycle, usability, test-driven development, the future of Drupal and the semantic web, etc. There is a lot of material in this presentation and during the course of the next few weeks, I plan to decompose this presentation in a number of extended blog posts. Stay tuned!

From infinite extensibility to infinite interoperability

Jeff Whatcott, Acquia's VP of marketing, wrote an interesting blog post about the term content management system, Microsoft SharePoint 2007, and Drupal's opportunity to be the poster child for the social software market space.

The funny thing is that I replied to Jeff's post back in 2006. In 2006, I agreed with Jeff -- and I still agree with Jeff today -- that (i) the term 'content management system' under sells what Drupal is capable of, (ii) that content management systems are consolidating to community and collaboration platforms, and (iii) that SharePoint is mind-boggling in more than one way.

From a content management system's point of view, we can summarize the current state of affairs as:

Web 1.0 = content management
Web 2.0 = Web 1.0 + user management + infinite extensibility

Jeff said it best when he wrote: the genius of what the [Drupal] community has done is to reduce all of the aspects of social software to their core DNA: content nodes and membership, and then build a platform that could be infinitely extended to allow the assembly of almost any styles of online social interaction.

But while Jeff rightfully sees a business opportunity for the Drupal community in the social publishing market, I tend to worry more about the fact that Drupal's key differentiator (i.e. bundling a wide variety of functionality into a single platform) becomes a commodity.

I want the Drupal community to stay ahead of the competition. I want to start implementing today what proprietary CMS vendors will implement in 2013. From a content management system's point of view, I believe, that means (and I really hate to use the term 'Web 3.0'):

Web 3.0 = Web 2.0 + infinite interoperability

which roughly translates to:

Web 3.0 = Web 2.0 + data portability + web service APIs

While the short-term business opportunity might be to go after the social publishing market, I strongly believe that the long-term business opportunity lies in the infinite interoperability and that spans well beyond the social software market.

Thanks to Open Source software and companies like Google, the cost of building Web 2.0 applications will approach zero. Contrary to what one might think, this actually creates a lot of business opportunities. Opportunities that are best monetized through web services. But for that to happen, ubiquitous and seamless interoperability is key.

State of Drupal presentation (September 2007)

Robert Garrigos from the Drupal Catalan user group just uploaded a video of the State of Drupal presentation that I gave at DrupalCon Barcelona 3 months ago. The presentation shares the results of a survey that I conducted in August; the survey ran for 30+ days, collected more than 1000 responses, and should do a pretty good job capturing the Drupal Zeitgeist.

Have a look at the video below (my presentation starts 12 minutes into the video) and/or download a copy of my slides (PDF, 14MB).

The future of snail mail

A video in Dutch about what we might tell our children about the world we lived in -- nostalgic memories of my childhood and why it is acceptable to close down more post offices. Via Bart De Waele.

Acquia, my Drupal startup

Big news today! I'm doing a Drupal startup.

The Drupal community does an incredible job building the Drupal technology and making sure that Drupal is on the forefront of the technical innovation. The Drupal Association does a great job supporting and protecting that community by improving our server infrastructure, by organizing Drupal conferences, by helping to protect the Drupal trademark. Last but not least, the Drupal consultants do an outstanding job developing websites and training people to use Drupal. Together we managed to create an incredibly successful project.

However, one piece is missing. Before we go there, let me provide a little more context.

I’ve been spending a lot of time thinking about the future of the web, and the future of Drupal in particular. I’ve also increasingly been spending time on what I want to do after I’m done with my PhD work. Since the two of those are coming together shortly, it’s time for me to start blogging about the next stages of Drupal, and my life.

The vision

First, Drupal 6 and Drupal 7 should be all about reaching out to more and different people. Making Drupal easier to use, easier to theme, easier to translate, and easier to develop for. Drupal 6 will do exactly that, and with Drupal 7 we should maintain that strategy. We want to make Drupal the best web content management platform; “the Linux of the web”.

The beauty of Drupal is that people can build powerful websites with little effort simply by combining different modules into one site; it is something we should continue to optimize for. I hope that that as a community you want to join me in putting (more of) the custom content types and (part of) views into Drupal core. But not just that; Drupal’s success arises from its community and the hundreds of contributed modules this community creates. We should continue to empower our contributors so they can continue to write great modules and deliver them in an exceptional way.

Second, with the rise of Facebook, Open Social, and friends, I’m not sure there is a future for (social community) websites that don’t provide an API. Starting with Drupal 7 we should also start to focus more on the ability to create, share and mash up managed content. The idea is to let Drupal be a data repository that can be accessed by tools and websites across the network. It is where custom content types, web service APIs, semantic web technologies and Drupal’s fine-grained access control mechanism come together. I want Drupal 7 to be a stepping stone when it comes to data mobility. This will allow people and companies to create value-added services that improve the users’ efficiency.

Third, when I examine the landscape of open source projects that have had big impact on the technology industry, I’ve concluded that projects which have had the biggest impact (usually) have a well-capitalized company behind them. Jboss, Linux and MySQL have all benefited not only technically, but the presence of a well-capitalized provider for those projects has made those projects palatable to users who might not have otherwise tried the software.

The company

So what is missing? It's two things: (i) a company that supports me in providing leadership to the Drupal community in exploring the vision I described above, and (ii) a company that is to Drupal what Ubuntu or RedHat are to Linux. If we want Drupal to grow by at least a factor of 10, keeping Drupal a hobby project as it is today, and taking a regular programming job at a big Belgian bank is clearly not going to cut it.

Thus, I'm starting a Drupal company whose current working name is 'Acquia'. Acquia's software products will include a number of Drupal distributions -- for community networks, digital media properties, corporate websites, and others. In addition to providing Drupal distributions, Acquia will build the Drupal-tuned analogue of the RedHat Network, over which we can deliver a wide variety of electronic services intended to be useful to people developing and operating Drupal websites. An example such service is an automated upgrade/update service, an uptime and performance monitoring / reporting service, a configuration management service, etc.

I was fortunate enough to meet an experienced CEO, Jay Batson, that I have come to like and trust, who managed to translate this vision into a business plan and who can complement my technical strength and community management skills with business experience in running open source software companies. (The last company Jay started was Pingtel, and open source enterprise-scale IP PBX, recently acquired by Bluesocket.) Jay has been invaluable so far.

The inevitable fear

Well, fear not.

Acquia is not going to fork or close-source Drupal. Acquia wants to see the Drupal community succeed and to do so, Acquia will listen to and work with the community to advance Drupal. The Drupal Association continues to operate the drupal.org domain, I continue to own the Drupal trademark, and the Drupal community continues to set the technical direction of the Drupal project. Drupal.com has not been sold.

Acquia's success is directly tied to overall success of the Drupal project - and to how widely-used it becomes. We understand better than anyone else that Acquia will never succeed on its own; we will only succeed if we are part of the larger Drupal community. We will contribute to Drupal development just as other companies or individuals do today. Our investors fully expect us to use a portion of the resources they’ve provided to help make Drupal even better, since our own success depends on significantly growing the widespread use of Drupal.

Furthermore, I'm expressly permitted to make decisions within the Drupal project that may not always be in Acquia’s best commercial interest. This was a hard requirement for me. Acquia fully expects that a portion of my time will be spent on activities associated with the project at large (vs. Acquia’s own software development). In essence, since the health and vitality of the Drupal project at large is extremely important to us, we’ve taken great pains to make sure that I am able to continue to act for the best interests of the Drupal community at large as I have done for the past 7 years.

The community has my heart and respect, and that won't change. Fear not.

Conclusions

So rather than working on Drupal in my spare time, I will soon have the time and resources to provide the leadership it takes to help get Drupal to the next level. I'm looking forward to leading the many thousands of you to the next step of this incredible adventure. It's been a little bit hard for me to not say anything about this before - mostly because I'm so excited about it. But it didn't make business sense to speak about this effort until it was for-real. Now that it is, I'm much happier that I can talk about it, because I want to think together with all of you about how we can make it a really really good thing for Drupal.

As a start, I've prepared a FAQ for our website. We (the company) will have more to say as we go along, and I will, too. Keep an eye on this blog or on our company website.