The future is a RESTful Drupal

Last weekend, we held a sprint at the Acquia offices for the Web Services and Context Core (WSCCI) Initiative for Drupal 8. This was an important sprint for the future of Drupal. This blog post provides a high-level overview of what was discussed and agreed upon; the different sprint participants will be laying out more technical details in follow-up blog posts.

Overall, a wide range of experience levels, skill sets, and perspectives were brought to the table, with the goal of the sprint being to clearly define the initiative’s scope, get agreement on what we wanted to accomplish and why, and lay out a clear plan for how to accomplish this.

WSCCI sprint group photo

In attendance were:

Scope

The WSCCI initiative, as envisioned by Larry Garfield, was originally set to address Drupal's web services and flexible page layout capabilities. We discovered that both would require significant changes to Drupal core, and it was difficult to build consensus online, so we decided to get together for 3 days and to flesh out what we actually wanted to accomplish, and how.

At the sprint, we first attempted to articulate all of the problems that WSCCI was trying to solve, which included: multiple page layouts, better UI/UX to manage blocks, partial page rendering (ESI, AHAH), contextual blocks, different response types per delivery callback/URL, plugin system / swappable subsystems, lazy loading bootstrap (convert subsystems to PSR-0 classes), infrastructure for building web services, dependency injection, and so on.

We then did a round of voting where we could each choose 3 of those things in order to try to determine which of those were the most important.

WSCCI sprint Post-it notes

Two things became instantly clear during this exercise:

  1. The items encompassed under WSCCI really spanned at least 3 separate major areas: Web Services, more robust ESI-based layouts (think Panels only more powerful), and cleaning up our underlying toolset to be a more loosely-coupled framework.
  2. The underlying architecture to support RESTful calls to Drupal that makes all of the other things possible was deemed the most important thing to focus on.

Scope resolution

After a good chunk of discussions, all were in agreement to scale back the scope of the initiative to just the "Web Services" piece, and spin off the Layout/blocks/related-UI parts to a separate effort.

Furthermore, some efforts, such as PSR-0 and Unified Plugin system, were only semi-related to the WSCCI initiative in the first place, and just happened to become relevant for it. Work on those efforts will continue as part of the general Framework community efforts.

Architecture

Fabien was able to offer a tremendous number of insights as to how various Symfony2 components could help provide underlying structure for Drupal core to support Web Services out of the box. Essentially, most of what the WSCCI team had been trying to work toward, in the abstract, was already implemented within Symfony2. While some implementation details were different than what we had in mind, the end result is almost exactly what we have been trying to accomplish. We therefore agreed that the best way forward was to leverage several Symfony2 components within Drupal as a way to speed progress toward that end.

Benefits

Some of the concrete benefits that would come out of these changes are

  • An underlying framework that is a first-class REST citizen. That means developing web services becomes easier and more efficient, because the plumbing is already in place. An HTML page is a particular case of a REST service.
  • Because the core system will be fully REST-ified, we'll be able to improve existing APIs and expose Drupal content in a natively RESTful way. For example, our current AJAX system doesn't comply with REST standards, but with these changes, can be cleaned up to do so.
  • That in turn makes Drupal-to-Drupal communication, content staging, content sharing, and a host of other related tasks easier.
  • The use of widely used libraries and techniques makes Drupal more approachable to new developers.

Why does this matter?

As it has evolved into an increasingly powerful system, Drupal has gotten increasingly complex and is not as easy to start developing with as it once was. Many developers are nervous about continuing that trend. Managing complexity is a challenge faced by many software projects, and one approach is to use standardized patterns and components.

Due to its long support for PHP 4, as well as some unique needs, Drupal does not take full advantage of standardized patterns and components. The complexity of the custom code that’s used and the non-standard architecture combines to create a barrier to entry for developers new to Drupal (both experienced and novice developers alike).

Meanwhile, the web is constantly changing around us. Web services and mobile are more important than ever, and with that comes the need to have more flexible page and layout capabilities. Now feels like the right time to modernize Drupal’s capabilities to bring it to the forefront of modern PHP systems and web systems in general.

While changing Drupal's core plumbing to address these needs, it's also a good opportunity to do so using more widely understood and modern techniques and libraries. Leveraging the work of a fellow open source community lets us get a jump on these changes to build a more powerful, more flexible, and more easily learnable system in less time than it would take to go it our own.

While these changes may seem large, and some of them are, we believe that they will achieve the right balance between empowering Drupal's design and architecture, and moving toward more modern, standard, well-tested code and techniques to empower a new generation of developers. I hope you are as excited as we are!

Comments

Nicolas Leroy (not verified):

Nice wrap-up. Thanks to the team for doing a great job to make Drupal 8 awesome!

February 16, 2012 - 21:12
Pol (not verified):

Really excited, can't wait to be there !

February 16, 2012 - 21:19
Damien McKenna (not verified):

Very pleased to see this, this is without a doubt the right direction for Drupal - have the easy & expansive CMS functionality along with a first-class framework, without having to write it all ourselves :)

February 16, 2012 - 22:58
Chris Johnson (not verified):

ESI? Are you talking about Varnish's Edge Side Includes language, or something else here?

February 17, 2012 - 00:42
Larry Garfield (not verified):

Yes, ESI as in Edge Side Includes, which apply to more than just Varnish. If you work from an assumption that the page being built in pieces and assembled in a caching layer (Varnish) or in the browser (hInclude, Backbone.js, etc.) is the normal case, and then emulate that in PHP as a "fallback degenerate case", you end up with a lot more power, flexibility, and cleanliness than if you treat partial page builds as the edge case.

Similarly, treating all incoming requests as a web service call, some of which just happen to be for HTML pages, you end up with a more robust, flexible, and powerful system than if you treat everything as an HTML page and everything else as an after-thought or add-on.

Drupal is moving to assuming the powerful modern stuff, and falling back to the simple stuff. This is a very good thing. :-)

February 17, 2012 - 05:47
Damien McKenna (not verified):

It should be noted that becoming RESTful is a major change that Ruby on Rails went through back in the day, so it's good to see Drupal finally taking the plunge: http://weblog.rubyonrails.org/2007/1/19/rails-1-2-rest-admiration-http-l...

February 17, 2012 - 02:54
Toan (not verified):

Wow, thanks for doing great job. you rock!

February 17, 2012 - 05:02
Carlo Mogavero (not verified):

This is great news.

I have been following this initiative the closest and can't wait for everything to be implemented as this plus EntityFieldQuery/DBTNG will enable CTools/Panels/Views like development out of the box.

February 17, 2012 - 06:14
Gábor Hojtsy (not verified):

With the "layout/blocks/related-ui" portion having a lot more direct user impact, it would be great to know how that spin-off is going to work? A new initiative?

February 17, 2012 - 10:02
EclipseGc (not verified):

Yes, Dries has asked me to head this up, and I am working out the details with my employer (Commerce Guys). Once I have answers on my end, things will become more official and announced, but I don't think I stepping on any toes saying this much. People in #drupal-wssci are already aware of the goings-on in regard to this, so I can't see the harm in saying it more publicly.

I have some very definitive ideas about how we should stage this, but I'll detail that once I know that I'm the person in charge of it.

Kris

February 17, 2012 - 16:48
Niels van Mourik (not verified):

1 up....

This is certainly the right direction for Drupal and would
re-empower core as a viable development framework up for
competition with Django, Cake and all the other cool stuff
out there.

I've been working on a boxed content-staging distribution
which showed me the importance of a cleanly decoupled set
of API's with powerful building blocks underneath it. The
great thing about this movement is not only that Drupal
would become a first-class REST/XML-RPC/SOAP platform,
it's has a very powerful content management system on
top of this out of the box!

Having this "one-step-ahead" on the web and improving it
with major steps like these will certainly boost Drupal
to remain being the killer-web-engine (TM).

Great initiatives!!

February 17, 2012 - 10:53
LevelFive Solutions (not verified):

Wow. This is what we want in Drupal. Thanks for the great team.

February 17, 2012 - 11:27
juliendsv (not verified):

Great news! I believe it's the right direction to go

February 17, 2012 - 11:31
John_B (not verified):

Very exciting, and surely the right direction for Drupal.

I'm not sure about combining WSCII with MVP. For clients and users small and large, Drupal is a love affair which turns out to be more expensive and demanding than people understand at the start. This makes it harder to explain to clients how much doing Drupal really costs http://www.mediacurrent.com/blog/understanding-pricing-around-drupal-ser..., as well as an army of confused newbies on the d.o forums.

Would Drupal be better software and better business if it did not aim to cover such a wide segment? WSCCI is great. However, high hosting costs, lack of backward compatibility, and complexity which new users underestimate are not going away. Can we say frankly that Drupal is the best CMS, but much as we would love to see it happen, Drupal should not try to combine this sophistication with being a suitable tool for users who want a one-click website?

February 17, 2012 - 11:36
Larry Garfield (not verified):

I'm not sure what you mean by "combining WSCCI with MVP". Minimum Viable Product?

I firmly believe that making Drupal a better tool for clicky-site-builders requires, first, making it a better underlying framework. "Just" making a clicky-tool leads to a single-purpose, brittle, inflexible clicky-tool. A clicky-tool built with a robust, flexible framework underneath leads to a more powerful clicky-tool.

That said, Drupal is not, nor has it ever been, nor really can it ever be, a "one-click website". That's just not its nature. You can use Drupal to BUILD a one-click website, such as Acquia Gardens, but Drupal core itself should never strive to be that. It's too limiting.

February 17, 2012 - 19:17
Jason (not verified):

I see no problem, Apps will take care of those one-click website users. And it sure to be more suitable than current situation.

And if you noticed, when someone start using a lot of apps. They don't feel like using much website anymore. IMO the end of static website is quite near. I think when Windows 8 out there will be a lot impact about this, especially from mobiles and tablets growth rate.

And we need cutting edge techs to handle these coming apps era.

Thanks

February 18, 2012 - 16:46
medios (not verified):

That is simply amazing!, Drupal is going to be much more powerful and versatile!. Thanks to the team for this great tools.

February 18, 2012 - 00:41
webchick (not verified):

For those who are new to Symfony, I found Fabien's 12-part blog series which starts at http://fabien.potencier.org/article/50/create-your-own-framework-on-top-... extremely helpful and approachable in illustrating both how the Symfony components work, as well as the concrete benefits of the RESTful approach. Definitely worth a read this weekend. :)

February 18, 2012 - 08:15
Christopher Cato (not verified):

Fantastic initiative and surely a great investment for the future of Drupal. Can't wait to see this in action. Keep it up guys!

February 18, 2012 - 12:52
uber (not verified):

Will it be damn easy to upgrade 5x or 6x to this new stuff?

Will users on all Drupal sites on different servers and different hosts be able to send messages or befriend one another like diaspora or status.net?

Will I as a webmaster of a Drupal site be easily able to allow any or all site on web to put a digg or like button towards my site?

Will end users be easily able to add any content or user like they do in google plus or facebook?

February 18, 2012 - 18:27

Add new comment

© 1999-2014 Dries Buytaert Creative Commons Attribution-NonCommercial-ShareAlike 3.0 License.
Drupal is a Registered Trademark of Dries Buytaert.