Making Drupal 8 fly

In my travels to talk about Drupal, everyone asks me about Drupal 8's performance and scalability. Modern websites are much more dynamic and interactive than 10 years ago, making it more difficult to build modern sites while also being fast. It made me realize that maybe I should write up a summary of some of the most exciting performance and scalability improvements in Drupal 8. After all, Drupal 8 will leapfrog many of its competitors in terms of how to architect and scale modern web applications. Many of these improvements benefit both small and large websites, but also allow us to build even bigger websites with Drupal.

More precise cache invalidation

One of the strategies we employ in making Drupal fast is "caching". This means we try to generate pages or page elements one time and then store them so future requests for those pages or page elements can be served faster. If an item is already cached, we can simply grab it without going through the building process again (known as a "cache hit"). Drupal stores each cache item in a "cache bin" (a database table, Memcache object, or whatever else is appropriate for the cache backend in use).

In Drupal 7 and before, when one of these cache items changes and it needs to be re-generated and re-stored (the cache gets "invalidated"), you can only delete a specific cache item, clear an entire cache bin, or use prefix-based invalidation. None of these three methods allow you to invalidate all cache items that contain data of, say, user 200. The only method that is going to suffice is clearing the entire cache bin, and this means that usually we invalidate way too much, resulting in poor cache hit ratios and wasted effort rebuilding cache items that haven't actually changed.

This problem is solved in Drupal 8 thanks to the concept of "cache tags": each cache item can have any number of cache tags. A cache tag is a compact string that describes the object being cached. Thanks to this extra metadata, we can now delete all cache items that use the user:200 cache tag, for example. This means we've deleted all the cache items we must delete, but not a single one more: optimal cache invalidation!

Drupal cache tags

Example cache tags for different cache IDs.

And don't worry, we also made sure to expose the cache tags to reverse proxies, so that efficient and accurate invalidation can happen throughout a site's entire delivery architecture.

More precise cache variation

While accurate cache invalidation makes caching more efficient, there is more we did to improve Drupal's caching. We also make sure that cached items are optimally varied. If you vary too much, duplicate cache entries will exist with the exact same content, resulting in inefficient usage of caches (low cache hit ratios). For example, we don't want a piece of content to be cached per user if it is the same for many users. If you vary too little, users might see incorrect content as two different cache entries might collide. In other words, you don't want to vary too much nor too little.

In Drupal 7 and before, it's easy to program any cached item to vary by user, by user role, and/or by page, and could even be configured through the UI for blocks. However, more targeted variations (such as by language, by country, or by content access permissions) were more difficult to program and not typically exposed in a configuration UI.

In Drupal 8, we introduced a Cache Context API to allow developers and site builders to express these variations and to make them automatically available in the configuration UI.

Drupal cache contexts

Server-side dynamic content substitution

Usually a page can be cached almost entirely except for a few dynamic elements. Often a page served to two different authenticated users looks identical except for a small "Welcome $name!" and perhaps their profile picture. In Drupal 7, this small personalization breaks the cacheability of the entire page (or rather, requires a cache context that's way too granular). Most parts of the page, like the header, the footer and certain blocks in the sidebars don't change often nor vary for each user, so why should you regenerate all those parts at every request?

In Drupal 8, thanks to the addition of #post_render_cache, that is no longer the case. Drupal 8 can render the entire page with some placeholder HTML for the name and profile picture. That page can then be cached. When Drupal has to serve that page to an authenticated user, it will retrieve it from the cache, and just before sending the HTML response to the client, it will substitute the placeholders with the dynamically rendered bits. This means we can avoid having to render the page over and over again, which is the expensive part, and only render those bits that need to be generated dynamically!

Client-side dynamic content substitution

Some things that Drupal has been rendering for the better part of a decade, such as the "new" and "updated" markers on comments, have always been rendered on the server. That is not ideal because these markers are different for every visitor and as a result, it makes caching pages with comments difficult.

The just-in-time substitution of placeholders with dynamic elements that #post_render_cache provides us can help address this. In some cases, as is the case with the comment markers, we can even do better and offload more work from the server to the client. In the case for comment markers, a certain comment is posted at a certain time — that doesn't vary per user. By embedding the comment timestamps as metadata in the DOM with a data-comment-timestamp="1424286665" attribute, we enable client-side JavaScript to render the comment markers, by fetching (and caching on the client side) the “last read" timestamp for the current user and simply comparing these numbers. Drupal 8 provides some framework code and API to make this easy.

A "Facebook BigPipe" render pipeline

With Drupal 8, we're very close to taking the client-side dynamic content substitution a step further, just like some of the world's largest dynamic websites do. Facebook has 1.35 billion monthly active users all requesting dynamic content, so why not learn from them?

The traditional page serving model has not kept up with the increase of highly personalized websites where different content is served to different users. In the traditional model, such as Drupal 7, the entire page is generated before it is sent to the browser: while Drupal is generating a page, the browser is idle and wasting its cycles doing nothing. When Drupal finishes generating the page and sends it to the browser, the browser kicks into action, and the web server is idle. In the case of Facebook, they use BigPipe. BigPipe delivers pages asynchronously instead; it parallelizes browser rendering and server processing. Instead of waiting for the entire page to be generated, BigPipe immediately sends a page skeleton to the the client so it can start rendering that. Then the remaining content elements are requested and injected into their correct place. From the user's perspective the page is rendered progressively. The initial page content becomes visible much earlier, which improves the perceived speed of the site.

We've made significant improvements to the way Drupal 8 renders pages (presentation). By default, Drupal 8 core still implements the traditional model of assembling these pieces into a complete page in a single server-side request, but the independence of each piece and the architecture of the new rendering pipeline enable different “render strategies" to be experimented with — different methods for dynamic content assembly, such as BigPipe, Edge Side Includes, or other ideas for making the most optimal use of client, server, content delivery networks and reverse proxies. In all those examples, the idea is that we can send the primary content first so the client can start rendering that. Then we send the remaining Drupal blocks, such as the navigation menu or a 'Related articles' block, and have the browser, content delivery network or reverse proxy assemble or combine these blocks into a page.

Drupal render pipeline

A snapshot of the Drupal 8 render pipeline diagram that highlights where alternative render strategies can be implemented.

Some early experiments by Wim Leers in Acquia's OCTO show that we can improve performance by a factor of about 2 compared to a recent Drupal 8 development snapshot. These breakthroughs are enabled by leveraging the various improvements we made to Drupal 8.

And much more

But that is not all. The Drupal community has actually done much more, including: complete asset dependency information (which allowed us to ensure zero JavaScript is loaded by default for anonymous users and send less data on AJAX requests), pluggable CSS/JS aggregation and minification (to support more optimal optimization algorithms), and more. We've also made sure Drupal 8 is fast by default, by having better defaults: CSS/JS aggregation enabled, JS assets being loaded from the bottom, block caching enabled, and so on.

All in all, there is a lot to look forward to in Drupal 8!

Special thanks to Acquia's Wim Leers, Alex Bronstein and Angie Byron for their contributions to this blog post.

Growing Drupal in Latin America

When I visited Brazil in 2011, I was so impressed by the Latin American Drupal community and how active and passionate the people are. The region is fun and beautiful, with some of the most amazing sites I have seen anywhere in the world. It also happens to be a strategic region for the project.

Latin American community members are doing their part to grow the project and the Drupal community. In 2014, the region hosted 19 Global Training Day events to recruit newcomers, and community leaders coordinated many Drupal camps to help convert those new Drupal users into skilled talent. Members of the Latin American community help promote Drupal at local technology and Open Source events, visiting events like FISL (7,000+ participants), Consegi (5,000+ participants) and Latinoware (4,500+ participants).

You can see the results of all the hard work in the growth of the Latin American Drupal business ecosystem. The region has a huge number of talented developers working at agencies large and small. When they aren't creating great Drupal websites like the one for the Rio 2016 Olympics, they are contributing code back to the project. For example, during our recent Global Sprint Weekend, communities in Bolivia, Colombia, Costa Rica, and Nicaragua participated and made valuable contributions.

The community has also been instrumental in translation efforts. On localize.drupal.org, the top translation is Spanish with 500 contributors, and a significant portion of those contributors come from the Latin America region. Community members are also investing time and energy translating Drupal educational videos, conducting camps in Spanish, and even publishing a Drupal magazine in Spanish. All of these efforts lower the barrier to entry for Spanish speakers, which is incredibly important because Spanish is one of the top spoken languages in the world. While the official language of the Drupal project is English, there can be a language divide for newcomers who primarily speak other languages.

Last but not least, I am excited that we are bringing DrupalCon to Latin America next week. This is the fruit of many hours spent by passionate volunteers in the Latin American local communities, working together with the Drupal Association to figure out how to make a DrupalCon happen in this part of the world. At every DrupalCon we have had so far, we have seen an increase in energy for the project and a bump in engagement. Come for the software, stay for the community! Hasta pronto!

On the hard choices we make every day

Every morning when I wake up, I have choices to make. While I want to turn Acquia into a billion dollar company, I also want to grow Drupal to be the leading digital experience platform. Both are connected and some of the work overlaps, but it still requires me to decide how much of my energy to focus on my duties as the CTO of Acquia as well as my duties as the project lead of Drupal.

It has been a few years since I wrote a good amount of code and I miss the thrill of programming -- both roles with Drupal and Acquia have evolved into management positions. Going back to writing software is a choice too, and one that I would undoubtedly enjoy. I think about it almost daily, and every time I decide not to.

At the same time, I also want to say 'yes' to the many invitations to travel the world, to speak at conferences, or to spend time with people I look up to. I also want to reply to all the emails I receive; I don't like it when emails fall through the cracks. I want to use my network and experience to advise other startups and Open Source projects. I'd love to increase my responsibilities as a Young Global Leader at the World Economic Forum (I'm bummed I couldn't attend Davos last week) and contribute to solving some of humanity's biggest problems. Other times I ask myself; why not kick back and have more time with friends and family? That is really important too.

When I push open the drapes in the morning, I have choices to make. The choices looked simpler when I was younger — most days I don't remember having to make choices at all. But as my work has grown in reach and impact, the choices in front of me have expanded as well. Every day, I struggle with these choices and ask myself how to spread my energy. I realize I'm not alone, as I know many others that have tough choices to make.

My guiding principle is to optimize for impact, purpose and passion — it is a delicate and personal balance based on the belief that somehow all the dots will connect.

My deep-wired desire to optimize for impact has not been without challenges. It has been an extremely strong force pulling me away from other relative priorities involving family, friends and personal health. Recently, I've gotten better at making time for family, friends, eating well and exercising. There is no denying that every decision has trade-offs: when I choose to do one thing it means I choose not do something else. Not doing something means I let people down, and as more and more choices present itself over time, it means letting down more and more people as well. If I let you down, I hope you understand. And one of the people I let down is myself, as I may never write software myself again -- it may never be the most impactful to do.

The best thing a human being can do is to help another human being. The organizations I'm building, the things I'm passionate about, the things I read about and the decisions I make will hopefully all lead to helping many more people. In turn, I hope that some of the people I have coached and worked with will pay it forward. Making choices is difficult but all in all, it's a wonderful feeling to see how many people I've touched by doing what I enjoy and love.

I am what I read

Almost every night before bed, I spend time reading. I love the feeling of falling asleep a little wiser than when I woke up. And often I read in the morning too. I read about photography, technology, investing, business, and more.

I love reading anything that provides a "mental workout"; articles or blog posts that stretch my mind or that point me in a different direction, that help me articulate my own emotions and my thoughts, or that make my imagination travel in time and space. Changing minds is changing lives. Reading makes me who I am.

While reading begets more reading, I love writing down my own thoughts as well, and that is what I decided to do this night. Sleep well!

Drupal retrospective 2014

It's that time again. Time to look back at 2014, and to look forward to 2015. For Drupal in 2014, it was all about Drupal 8. As Drupal 8's development enters its fourth (and hopefully, final) year of development, it's a good time to reflect on all the work achieved by the Drupal 8 team so far, and to talk about Drupal 8's momentum heading into the final stretch to the release.

Drupal 8 will have 200 new features. Among the larger features that I'm excited about are the responsive design, HTML5 support, the native web service support, the much improved multilingual support, the configuration management system, a built-in WYSIWYG editor, in-place editing, streamlined content editing, the improved entity system, and more. The list of improvements is long!

My favorite part of Drupal 8 is that it will make building all types of Drupal sites — both big and small — much easier than with Drupal 7.

Key accomplishments in 2014 include:

Drupal 8 beta 1 released

October 1, 2014, amidst the fanfare at DrupalCon Amsterdam, we released Drupal 8 beta 1. This was an important milestone in the project, marking the finalization of major APIs, which enables contributed modules to begin porting in earnest.

Total number of Drupal 8 contributors surpasses 2,500

Our 2,500th core contributor was Tasya Rukmana (tadityar), a high-school student participating in Google Code-in 2014! Awesome.

Kick-starting contributed modules in Drupal 8

Drupal 8's new object-oriented API represents a significant paradigm shift for developers (there are many benefits to this). To help Drupal 7 pros make the jump to Drupal 8, Acquia funded the Drupal Module Upgrader project. This project will not only scan a Drupal 7 module and generate a report pointing off to the appropriate documentation on how to port it, there is even a mode that automatically re-writes much of your module's code to Drupal 8 to eliminate a huge chunk of the work.

Sprints, sprints and more sprints!

We organized dozens of sprints all around the world, and together hundreds of people came together in "real life" to help get Drupal 8 released. Sprints are a key part of momentum-building in Drupal, by laser-focusing on a specific goal, or by pairing both new and experienced contributors together for mentorship. Not only do sprints make solving tough issues easier, they also provide opportunities for building relationships and "leveling up" your skills.

Drupal 8 accelerate fund

Though it was launched just a month ago, the Drupal Association's Drupal 8 Accelerate Fund is already helping to add velocity to Drupal 8, by paying key contributors to help fix particularly onerous critical issues.

What is in store for 2015?

Getting the Drupal 8 release done

Our current focus is resolving the Drupal 8 upgrade path issues, which will allow early adopters of Drupal 8 to upgrade their site data between beta releases, and should result in a further uptick to Drupal 8 development velocity.

Once we reach zero critical issues, we begin the release candidate phase. Among the areas left to polish up after the Drupal 8 upgrade path issues are bringing external libraries up to date, finalizing documentation, and performance.

Continuous improvements after Drupal 8

Unlike prior versions of Drupal, Drupal 8 has adopted a new release cycle that will provide backwards-compatible "feature" releases every 6 months. I'm extremely excited about this change, as it means we can innovate on the core platform for years to come after release, versus holding all of the new goodies until Drupal 9.

Getting more organizations to contribute

We're now one of the largest Open Source projects in terms of active contributors, if not the largest. That growth requires us to evolve how we work. Over the years, we've grown from a 100% volunteer-driven model to a model where there is a mix of volunteers, contributors who are partially funded by their customers or employers, and contributors who are paid full-time to work on Drupal.

While this shift has big benefits in making Drupal more sustainable, it also means there is increasingly more corporate participation and influence. One of our biggest challenges for 2015 is to figure out how we can get more commercial organizations to step up to take on more of the shared maintenance of Drupal, while at the same time respecting the needs and desires of our entire community.

Improving our governance model

There has also been a lot of talk about optimizing the way in which we work, to make it more explicit who is responsible for what, how decisions are made, and so on. This year I plan to work with others in the community to revamp Drupal core's governance model to bring more transparency and appoint additional leadership.

Conclusion

Overall, I'm thrilled with the progress that the Drupal core contributors have made in 2014, and want to extend an enormous thanks to each and every one of our 2,500 contributors who have brought us this far. I'm feeling very positive about our momentum going into 2015.

Drupal 8 will set a new standard for ease of use, power and flexibility, and will have something for everyone to love. Without a doubt, Drupal 8 will take our community to new heights. Let's do this!

Pages

© 1999-2015 Dries Buytaert Creative Commons Attribution-NonCommercial-ShareAlike 3.0 License.
Drupal is a Registered Trademark of Dries Buytaert.