I gave my State of Drupal presentation at DrupalCon Los Angeles in front of 3,000+ attendees. In case you didn't attend DrupalCon Los Angeles, you can watch the recording of my keynote or download a copy of my slides (PDF, 77 MB).
In the first part of the keynote, I talked about the history of the Drupal project, some of the challenges we overcame, and some of the lessons learned. While I have talked about our history in the past, it had been 6 years ago at DrupalCon Washington DC in 2009. In those 6 years, the Drupal community has grown so large that most people in the community don't know where we came from. Understanding the history of Drupal is important; it explains our culture, it holds us together in challenging times and provides a compass for where we are heading.
In the middle part of the keynote, I talked about what I believe is one of our biggest challenges; motivating more organizations to contribute more meaingfully to Drupal's development. Just as it is important to understand the history of Drupal, talking about the present is an important foundation for everyone in the community. It is hard to grow without the context of our current state.
In the third and last part of the keynote, I looked forward, talked about my vision for the big reverse of the web and how it relates to Drupal. The way the web is evolving provides us an opportunity to better understand our sites visitors or users and to build one-to-one relationships, something that much of our society has lost with the industrial revolution. If the web evolves the way I think it will, it will be both life changing and industry changing. While it won't be without concerns, we have a huge opportunity ahead of us, and Drupal 8 will help us build towards that future.
I'm proud of where we came from and excited for where we are headed. Take a look at the keynote if you want to learn more about it.
Earlier this week Matt Mullenweg, founder and CEO of Automattic, parent company of WordPress.com, announced the acquisition of WooCommerce. This is a very interesting move that I think cements the SMB/enterprise positioning between WordPress and Drupal.
As Matt points out a huge percentage of the digital experiences on the web are now powered by open source solutions: WordPress, Joomla and Drupal. Yet one question the acquisition may evoke is: "How will open source platforms drive ecommerce innovation in the future?".
Larger retailers with complex requirements usually rely on bespoke commerce engines or built their online stores on solutions such as Demandware, Hybris and Magento. Small businesses access essential functions such as secure transaction processing, product information management, shipping and tax calculations, and PCI compliance from third-party solutions such as Shopify, Amazon's merchant services and increasingly, solutions from Squarespace and Wix.
I believe the WooCommerce acquisition by Automattic puts WordPress in a better position to compete against the slickly marketed offerings from Squarespace and Wix, and defend WordPress's popular position among small businesses. WooCommerce brings to WordPress a commerce toolkit with essential functions such as payments processing, inventory management, cart checkout and tax calculations.
Drupal has a rich library of commerce solutions ranging from Drupal Commerce -- a library of modules offered by Commerce Guys -- to connectors offered by Acquia for Demandware and other ecommerce engines. Brands such as LUSH Cosmetics handle all of their ecommerce operations with Drupal, others, such as Puma, use a Drupal-Demandware integration to combine the best elements of content and commerce to deliver stunning shopping experiences that break down the old division between brand marketing experiences and the shopping process. Companies such as Tesla Motors have created their own custom commerce engine and rely on Drupal to deliver the front-end customer experience across multiple digital channels from traditional websites to mobile devices, in-store kiosks and more.
To me, this further accentuates the division of the CMS market with WordPress dominating the small business segment and Drupal further solidifying its position with larger organizations with more complex requirements. I'm looking forward to seeing what the next few years will bring for the open source commerce world, and I'd love to hear your opinion in the comments.
In my travels to talk about Drupal, everyone asks me about Drupal 8's performance and scalability. Modern websites are much more dynamic and interactive than 10 years ago, making it more difficult to build modern sites while also being fast. It made me realize that maybe I should write up a summary of some of the most exciting performance and scalability improvements in Drupal 8. After all, Drupal 8 will leapfrog many of its competitors in terms of how to architect and scale modern web applications. Many of these improvements benefit both small and large websites, but also allow us to build even bigger websites with Drupal.
More precise cache invalidation
One of the strategies we employ in making Drupal fast is "caching". This means we try to generate pages or page elements one time and then store them so future requests for those pages or page elements can be served faster. If an item is already cached, we can simply grab it without going through the building process again (known as a "cache hit"). Drupal stores each cache item in a "cache bin" (a database table, Memcache object, or whatever else is appropriate for the cache backend in use).
In Drupal 7 and before, when one of these cache items changes and it needs to be re-generated and re-stored (the cache gets "invalidated"), you can only delete a specific cache item, clear an entire cache bin, or use prefix-based invalidation. None of these three methods allow you to invalidate all cache items that contain data of, say, user 200. The only method that is going to suffice is clearing the entire cache bin, and this means that usually we invalidate way too much, resulting in poor cache hit ratios and wasted effort rebuilding cache items that haven't actually changed.
This problem is solved in Drupal 8 thanks to the concept of "cache tags": each cache item can have any number of cache tags. A cache tag is a compact string that describes the object being cached. Thanks to this extra metadata, we can now delete all cache items that use the
user:200 cache tag, for example. This means we've deleted all the cache items we must delete, but not a single one more: optimal cache invalidation!
And don't worry, we also made sure to expose the cache tags to reverse proxies, so that efficient and accurate invalidation can happen throughout a site's entire delivery architecture.
More precise cache variation
While accurate cache invalidation makes caching more efficient, there is more we did to improve Drupal's caching. We also make sure that cached items are optimally varied. If you vary too much, duplicate cache entries will exist with the exact same content, resulting in inefficient usage of caches (low cache hit ratios). For example, we don't want a piece of content to be cached per user if it is the same for many users. If you vary too little, users might see incorrect content as two different cache entries might collide. In other words, you don't want to vary too much nor too little.
In Drupal 7 and before, it's easy to program any cached item to vary by user, by user role, and/or by page, and could even be configured through the UI for blocks. However, more targeted variations (such as by language, by country, or by content access permissions) were more difficult to program and not typically exposed in a configuration UI.
In Drupal 8, we introduced a Cache Context API to allow developers and site builders to express these variations and to make them automatically available in the configuration UI.
Server-side dynamic content substitution
Usually a page can be cached almost entirely except for a few dynamic elements. Often a page served to two different authenticated users looks identical except for a small "Welcome $name!" and perhaps their profile picture. In Drupal 7, this small personalization breaks the cacheability of the entire page (or rather, requires a cache context that's way too granular). Most parts of the page, like the header, the footer and certain blocks in the sidebars don't change often nor vary for each user, so why should you regenerate all those parts at every request?
In Drupal 8, thanks to the addition of #post_render_cache, that is no longer the case. Drupal 8 can render the entire page with some placeholder HTML for the name and profile picture. That page can then be cached. When Drupal has to serve that page to an authenticated user, it will retrieve it from the cache, and just before sending the HTML response to the client, it will substitute the placeholders with the dynamically rendered bits. This means we can avoid having to render the page over and over again, which is the expensive part, and only render those bits that need to be generated dynamically!
Client-side dynamic content substitution
Some things that Drupal has been rendering for the better part of a decade, such as the "new" and "updated" markers on comments, have always been rendered on the server. That is not ideal because these markers are different for every visitor and as a result, it makes caching pages with comments difficult.
The just-in-time substitution of placeholders with dynamic elements that
#post_render_cache provides us can help address this. In some cases, as is the case with the comment markers, we can even do better and offload more work from the server to the client. In the case for comment markers, a certain comment is posted at a certain time — that doesn't vary per user. By embedding the comment timestamps as metadata in the DOM with a
A "Facebook BigPipe" render pipeline
With Drupal 8, we're very close to taking the client-side dynamic content substitution a step further, just like some of the world's largest dynamic websites do. Facebook has 1.35 billion monthly active users all requesting dynamic content, so why not learn from them?
The traditional page serving model has not kept up with the increase of highly personalized websites where different content is served to different users. In the traditional model, such as Drupal 7, the entire page is generated before it is sent to the browser: while Drupal is generating a page, the browser is idle and wasting its cycles doing nothing. When Drupal finishes generating the page and sends it to the browser, the browser kicks into action, and the web server is idle. In the case of Facebook, they use BigPipe. BigPipe delivers pages asynchronously instead; it parallelizes browser rendering and server processing. Instead of waiting for the entire page to be generated, BigPipe immediately sends a page skeleton to the the client so it can start rendering that. Then the remaining content elements are requested and injected into their correct place. From the user's perspective the page is rendered progressively. The initial page content becomes visible much earlier, which improves the perceived speed of the site.
We've made significant improvements to the way Drupal 8 renders pages (presentation). By default, Drupal 8 core still implements the traditional model of assembling these pieces into a complete page in a single server-side request, but the independence of each piece and the architecture of the new rendering pipeline enable different “render strategies" to be experimented with — different methods for dynamic content assembly, such as BigPipe, Edge Side Includes, or other ideas for making the most optimal use of client, server, content delivery networks and reverse proxies. In all those examples, the idea is that we can send the primary content first so the client can start rendering that. Then we send the remaining Drupal blocks, such as the navigation menu or a 'Related articles' block, and have the browser, content delivery network or reverse proxy assemble or combine these blocks into a page.
Some early experiments by Wim Leers in Acquia's OCTO show that we can improve performance by a factor of about 2 compared to a recent Drupal 8 development snapshot. These breakthroughs are enabled by leveraging the various improvements we made to Drupal 8.
And much more
All in all, there is a lot to look forward to in Drupal 8!
When I visited Brazil in 2011, I was so impressed by the Latin American Drupal community and how active and passionate the people are. The region is fun and beautiful, with some of the most amazing sites I have seen anywhere in the world. It also happens to be a strategic region for the project.
Latin American community members are doing their part to grow the project and the Drupal community. In 2014, the region hosted 19 Global Training Day events to recruit newcomers, and community leaders coordinated many Drupal camps to help convert those new Drupal users into skilled talent. Members of the Latin American community help promote Drupal at local technology and Open Source events, visiting events like FISL (7,000+ participants), Consegi (5,000+ participants) and Latinoware (4,500+ participants).
You can see the results of all the hard work in the growth of the Latin American Drupal business ecosystem. The region has a huge number of talented developers working at agencies large and small. When they aren't creating great Drupal websites like the one for the Rio 2016 Olympics, they are contributing code back to the project. For example, during our recent Global Sprint Weekend, communities in Bolivia, Colombia, Costa Rica, and Nicaragua participated and made valuable contributions.
The community has also been instrumental in translation efforts. On localize.drupal.org, the top translation is Spanish with 500 contributors, and a significant portion of those contributors come from the Latin America region. Community members are also investing time and energy translating Drupal educational videos, conducting camps in Spanish, and even publishing a Drupal magazine in Spanish. All of these efforts lower the barrier to entry for Spanish speakers, which is incredibly important because Spanish is one of the top spoken languages in the world. While the official language of the Drupal project is English, there can be a language divide for newcomers who primarily speak other languages.
Last but not least, I am excited that we are bringing DrupalCon to Latin America next week. This is the fruit of many hours spent by passionate volunteers in the Latin American local communities, working together with the Drupal Association to figure out how to make a DrupalCon happen in this part of the world. At every DrupalCon we have had so far, we have seen an increase in energy for the project and a bump in engagement. Come for the software, stay for the community! Hasta pronto!
It's that time again. Time to look back at 2014, and to look forward to 2015. For Drupal in 2014, it was all about Drupal 8. As Drupal 8's development enters its fourth (and hopefully, final) year of development, it's a good time to reflect on all the work achieved by the Drupal 8 team so far, and to talk about Drupal 8's momentum heading into the final stretch to the release.
Drupal 8 will have 200 new features. Among the larger features that I'm excited about are the responsive design, HTML5 support, the native web service support, the much improved multilingual support, the configuration management system, a built-in WYSIWYG editor, in-place editing, streamlined content editing, the improved entity system, and more. The list of improvements is long!
My favorite part of Drupal 8 is that it will make building all types of Drupal sites — both big and small — much easier than with Drupal 7.
Key accomplishments in 2014 include:
Drupal 8 beta 1 released
October 1, 2014, amidst the fanfare at DrupalCon Amsterdam, we released Drupal 8 beta 1. This was an important milestone in the project, marking the finalization of major APIs, which enables contributed modules to begin porting in earnest.
Total number of Drupal 8 contributors surpasses 2,500
Kick-starting contributed modules in Drupal 8
Drupal 8's new object-oriented API represents a significant paradigm shift for developers (there are many benefits to this). To help Drupal 7 pros make the jump to Drupal 8, Acquia funded the Drupal Module Upgrader project. This project will not only scan a Drupal 7 module and generate a report pointing off to the appropriate documentation on how to port it, there is even a mode that automatically re-writes much of your module's code to Drupal 8 to eliminate a huge chunk of the work.
Sprints, sprints and more sprints!
We organized dozens of sprints all around the world, and together hundreds of people came together in "real life" to help get Drupal 8 released. Sprints are a key part of momentum-building in Drupal, by laser-focusing on a specific goal, or by pairing both new and experienced contributors together for mentorship. Not only do sprints make solving tough issues easier, they also provide opportunities for building relationships and "leveling up" your skills.
Drupal 8 accelerate fund
Though it was launched just a month ago, the Drupal Association's Drupal 8 Accelerate Fund is already helping to add velocity to Drupal 8, by paying key contributors to help fix particularly onerous critical issues.
What is in store for 2015?
Getting the Drupal 8 release done
Our current focus is resolving the Drupal 8 upgrade path issues, which will allow early adopters of Drupal 8 to upgrade their site data between beta releases, and should result in a further uptick to Drupal 8 development velocity.
Once we reach zero critical issues, we begin the release candidate phase. Among the areas left to polish up after the Drupal 8 upgrade path issues are bringing external libraries up to date, finalizing documentation, and performance.
Continuous improvements after Drupal 8
Unlike prior versions of Drupal, Drupal 8 has adopted a new release cycle that will provide backwards-compatible "feature" releases every 6 months. I'm extremely excited about this change, as it means we can innovate on the core platform for years to come after release, versus holding all of the new goodies until Drupal 9.
Getting more organizations to contribute
We're now one of the largest Open Source projects in terms of active contributors, if not the largest. That growth requires us to evolve how we work. Over the years, we've grown from a 100% volunteer-driven model to a model where there is a mix of volunteers, contributors who are partially funded by their customers or employers, and contributors who are paid full-time to work on Drupal.
While this shift has big benefits in making Drupal more sustainable, it also means there is increasingly more corporate participation and influence. One of our biggest challenges for 2015 is to figure out how we can get more commercial organizations to step up to take on more of the shared maintenance of Drupal, while at the same time respecting the needs and desires of our entire community.
Improving our governance model
There has also been a lot of talk about optimizing the way in which we work, to make it more explicit who is responsible for what, how decisions are made, and so on. This year I plan to work with others in the community to revamp Drupal core's governance model to bring more transparency and appoint additional leadership.
Overall, I'm thrilled with the progress that the Drupal core contributors have made in 2014, and want to extend an enormous thanks to each and every one of our 2,500 contributors who have brought us this far. I'm feeling very positive about our momentum going into 2015.
Drupal 8 will set a new standard for ease of use, power and flexibility, and will have something for everyone to love. Without a doubt, Drupal 8 will take our community to new heights. Let's do this!