You are here
In my travels to talk about Drupal, everyone asks me about Drupal 8's performance and scalability. Modern websites are much more dynamic and interactive than 10 years ago, making it more difficult to build modern sites while also being fast. It made me realize that maybe I should write up a summary of some of the most exciting performance and scalability improvements in Drupal 8. After all, Drupal 8 will leapfrog many of its competitors in terms of how to architect and scale modern web applications. Many of these improvements benefit both small and large websites, but also allow us to build even bigger websites with Drupal.
More precise cache invalidation
One of the strategies we employ in making Drupal fast is "caching". This means we try to generate pages or page elements one time and then store them so future requests for those pages or page elements can be served faster. If an item is already cached, we can simply grab it without going through the building process again (known as a "cache hit"). Drupal stores each cache item in a "cache bin" (a database table, Memcache object, or whatever else is appropriate for the cache backend in use).
In Drupal 7 and before, when one of these cache items changes and it needs to be re-generated and re-stored (the cache gets "invalidated"), you can only delete a specific cache item, clear an entire cache bin, or use prefix-based invalidation. None of these three methods allow you to invalidate all cache items that contain data of, say, user 200. The only method that is going to suffice is clearing the entire cache bin, and this means that usually we invalidate way too much, resulting in poor cache hit ratios and wasted effort rebuilding cache items that haven't actually changed.
This problem is solved in Drupal 8 thanks to the concept of "cache tags": each cache item can have any number of cache tags. A cache tag is a compact string that describes the object being cached. Thanks to this extra metadata, we can now delete all cache items that use the
user:200 cache tag, for example. This means we've deleted all the cache items we must delete, but not a single one more: optimal cache invalidation!
Example cache tags for different cache IDs.
And don't worry, we also made sure to expose the cache tags to reverse proxies, so that efficient and accurate invalidation can happen throughout a site's entire delivery architecture.
More precise cache variation
While accurate cache invalidation makes caching more efficient, there is more we did to improve Drupal's caching. We also make sure that cached items are optimally varied. If you vary too much, duplicate cache entries will exist with the exact same content, resulting in inefficient usage of caches (low cache hit ratios). For example, we don't want a piece of content to be cached per user if it is the same for many users. If you vary too little, users might see incorrect content as two different cache entries might collide. In other words, you don't want to vary too much nor too little.
In Drupal 7 and before, it's easy to program any cached item to vary by user, by user role, and/or by page, and could even be configured through the UI for blocks. However, more targeted variations (such as by language, by country, or by content access permissions) were more difficult to program and not typically exposed in a configuration UI.
In Drupal 8, we introduced a Cache Context API to allow developers and site builders to express these variations and to make them automatically available in the configuration UI.
Server-side dynamic content substitution
Usually a page can be cached almost entirely except for a few dynamic elements. Often a page served to two different authenticated users looks identical except for a small "Welcome $name!" and perhaps their profile picture. In Drupal 7, this small personalization breaks the cacheability of the entire page (or rather, requires a cache context that's way too granular). Most parts of the page, like the header, the footer and certain blocks in the sidebars don't change often nor vary for each user, so why should you regenerate all those parts at every request?
In Drupal 8, thanks to the addition of #post_render_cache, that is no longer the case. Drupal 8 can render the entire page with some placeholder HTML for the name and profile picture. That page can then be cached. When Drupal has to serve that page to an authenticated user, it will retrieve it from the cache, and just before sending the HTML response to the client, it will substitute the placeholders with the dynamically rendered bits. This means we can avoid having to render the page over and over again, which is the expensive part, and only render those bits that need to be generated dynamically!
Client-side dynamic content substitution
Some things that Drupal has been rendering for the better part of a decade, such as the "new" and "updated" markers on comments, have always been rendered on the server. That is not ideal because these markers are different for every visitor and as a result, it makes caching pages with comments difficult.
The just-in-time substitution of placeholders with dynamic elements that
#post_render_cache provides us can help address this. In some cases, as is the case with the comment markers, we can even do better and offload more work from the server to the client. In the case for comment markers, a certain comment is posted at a certain time — that doesn't vary per user. By embedding the comment timestamps as metadata in the DOM with a
A "Facebook BigPipe" render pipeline
With Drupal 8, we're very close to taking the client-side dynamic content substitution a step further, just like some of the world's largest dynamic websites do. Facebook has 1.35 billion monthly active users all requesting dynamic content, so why not learn from them?
The traditional page serving model has not kept up with the increase of highly personalized websites where different content is served to different users. In the traditional model, such as Drupal 7, the entire page is generated before it is sent to the browser: while Drupal is generating a page, the browser is idle and wasting its cycles doing nothing. When Drupal finishes generating the page and sends it to the browser, the browser kicks into action, and the web server is idle. In the case of Facebook, they use BigPipe. BigPipe delivers pages asynchronously instead; it parallelizes browser rendering and server processing. Instead of waiting for the entire page to be generated, BigPipe immediately sends a page skeleton to the the client so it can start rendering that. Then the remaining content elements are requested and injected into their correct place. From the user's perspective the page is rendered progressively. The initial page content becomes visible much earlier, which improves the perceived speed of the site.
We've made significant improvements to the way Drupal 8 renders pages (presentation). By default, Drupal 8 core still implements the traditional model of assembling these pieces into a complete page in a single server-side request, but the independence of each piece and the architecture of the new rendering pipeline enable different “render strategies" to be experimented with — different methods for dynamic content assembly, such as BigPipe, Edge Side Includes, or other ideas for making the most optimal use of client, server, content delivery networks and reverse proxies. In all those examples, the idea is that we can send the primary content first so the client can start rendering that. Then we send the remaining Drupal blocks, such as the navigation menu or a 'Related articles' block, and have the browser, content delivery network or reverse proxy assemble or combine these blocks into a page.
A snapshot of the Drupal 8 render pipeline diagram that highlights where alternative render strategies can be implemented.
Some early experiments by Wim Leers in Acquia's OCTO show that we can improve performance by a factor of about 2 compared to a recent Drupal 8 development snapshot. These breakthroughs are enabled by leveraging the various improvements we made to Drupal 8.
And much more
All in all, there is a lot to look forward to in Drupal 8!
When I visited Brazil in 2011, I was so impressed by the Latin American Drupal community and how active and passionate the people are. The region is fun and beautiful, with some of the most amazing sites I have seen anywhere in the world. It also happens to be a strategic region for the project.
Latin American community members are doing their part to grow the project and the Drupal community. In 2014, the region hosted 19 Global Training Day events to recruit newcomers, and community leaders coordinated many Drupal camps to help convert those new Drupal users into skilled talent. Members of the Latin American community help promote Drupal at local technology and Open Source events, visiting events like FISL (7,000+ participants), Consegi (5,000+ participants) and Latinoware (4,500+ participants).
You can see the results of all the hard work in the growth of the Latin American Drupal business ecosystem. The region has a huge number of talented developers working at agencies large and small. When they aren't creating great Drupal websites like the one for the Rio 2016 Olympics, they are contributing code back to the project. For example, during our recent Global Sprint Weekend, communities in Bolivia, Colombia, Costa Rica, and Nicaragua participated and made valuable contributions.
The community has also been instrumental in translation efforts. On localize.drupal.org, the top translation is Spanish with 500 contributors, and a significant portion of those contributors come from the Latin America region. Community members are also investing time and energy translating Drupal educational videos, conducting camps in Spanish, and even publishing a Drupal magazine in Spanish. All of these efforts lower the barrier to entry for Spanish speakers, which is incredibly important because Spanish is one of the top spoken languages in the world. While the official language of the Drupal project is English, there can be a language divide for newcomers who primarily speak other languages.
Last but not least, I am excited that we are bringing DrupalCon to Latin America next week. This is the fruit of many hours spent by passionate volunteers in the Latin American local communities, working together with the Drupal Association to figure out how to make a DrupalCon happen in this part of the world. At every DrupalCon we have had so far, we have seen an increase in energy for the project and a bump in engagement. Come for the software, stay for the community! Hasta pronto!
It's that time again. Time to look back at 2014, and to look forward to 2015. For Drupal in 2014, it was all about Drupal 8. As Drupal 8's development enters its fourth (and hopefully, final) year of development, it's a good time to reflect on all the work achieved by the Drupal 8 team so far, and to talk about Drupal 8's momentum heading into the final stretch to the release.
Drupal 8 will have 200 new features. Among the larger features that I'm excited about are the responsive design, HTML5 support, the native web service support, the much improved multilingual support, the configuration management system, a built-in WYSIWYG editor, in-place editing, streamlined content editing, the improved entity system, and more. The list of improvements is long!
My favorite part of Drupal 8 is that it will make building all types of Drupal sites — both big and small — much easier than with Drupal 7.
Key accomplishments in 2014 include:
Drupal 8 beta 1 released
October 1, 2014, amidst the fanfare at DrupalCon Amsterdam, we released Drupal 8 beta 1. This was an important milestone in the project, marking the finalization of major APIs, which enables contributed modules to begin porting in earnest.
Total number of Drupal 8 contributors surpasses 2,500
Kick-starting contributed modules in Drupal 8
Drupal 8's new object-oriented API represents a significant paradigm shift for developers (there are many benefits to this). To help Drupal 7 pros make the jump to Drupal 8, Acquia funded the Drupal Module Upgrader project. This project will not only scan a Drupal 7 module and generate a report pointing off to the appropriate documentation on how to port it, there is even a mode that automatically re-writes much of your module's code to Drupal 8 to eliminate a huge chunk of the work.
Sprints, sprints and more sprints!
We organized dozens of sprints all around the world, and together hundreds of people came together in "real life" to help get Drupal 8 released. Sprints are a key part of momentum-building in Drupal, by laser-focusing on a specific goal, or by pairing both new and experienced contributors together for mentorship. Not only do sprints make solving tough issues easier, they also provide opportunities for building relationships and "leveling up" your skills.
Drupal 8 accelerate fund
Though it was launched just a month ago, the Drupal Association's Drupal 8 Accelerate Fund is already helping to add velocity to Drupal 8, by paying key contributors to help fix particularly onerous critical issues.
What is in store for 2015?
Getting the Drupal 8 release done
Our current focus is resolving the Drupal 8 upgrade path issues, which will allow early adopters of Drupal 8 to upgrade their site data between beta releases, and should result in a further uptick to Drupal 8 development velocity.
Once we reach zero critical issues, we begin the release candidate phase. Among the areas left to polish up after the Drupal 8 upgrade path issues are bringing external libraries up to date, finalizing documentation, and performance.
Continuous improvements after Drupal 8
Unlike prior versions of Drupal, Drupal 8 has adopted a new release cycle that will provide backwards-compatible "feature" releases every 6 months. I'm extremely excited about this change, as it means we can innovate on the core platform for years to come after release, versus holding all of the new goodies until Drupal 9.
Getting more organizations to contribute
We're now one of the largest Open Source projects in terms of active contributors, if not the largest. That growth requires us to evolve how we work. Over the years, we've grown from a 100% volunteer-driven model to a model where there is a mix of volunteers, contributors who are partially funded by their customers or employers, and contributors who are paid full-time to work on Drupal.
While this shift has big benefits in making Drupal more sustainable, it also means there is increasingly more corporate participation and influence. One of our biggest challenges for 2015 is to figure out how we can get more commercial organizations to step up to take on more of the shared maintenance of Drupal, while at the same time respecting the needs and desires of our entire community.
Improving our governance model
There has also been a lot of talk about optimizing the way in which we work, to make it more explicit who is responsible for what, how decisions are made, and so on. This year I plan to work with others in the community to revamp Drupal core's governance model to bring more transparency and appoint additional leadership.
Overall, I'm thrilled with the progress that the Drupal core contributors have made in 2014, and want to extend an enormous thanks to each and every one of our 2,500 contributors who have brought us this far. I'm feeling very positive about our momentum going into 2015.
Drupal 8 will set a new standard for ease of use, power and flexibility, and will have something for everyone to love. Without a doubt, Drupal 8 will take our community to new heights. Let's do this!
As is now a tradition for me, here is my annual Acquia retrospective, where I look back at 2014 and share what is on my mind as we start the new year. I take the time to write these retrospectives not only for you dear reader, but also for myself, because I want to keep a record of the changes we've gone through as a company and how my personal thinking is evolving from year to year. But I also write them for you, because you might be able to learn from my experiences or from analyzing the information provided. If you would like to, you can read my previous retrospectives: 2009, 2010, 2011, 2012 and 2013.
For Acquia, 2014 was another incredible year, one where we beat our wildest expectations. We crossed the major milestone of $100 USD million in annual revenue, the majority of which is recurring subscription revenue. It is hard to believe that 2014 was only our sixth full year as a revenue-generating business.
We've seen the most growth from our enterprise customers, but our number of small and medium size customers has grown too. We helped launch and host some incredible sites last year: from weather.com (a top 20 site) to the Emmys. Our efforts in Europe and Asia-Pacific are paying off; our EMEA business grew substantially, and the Australian government decided to switch the entire government to Drupal and the Acquia Platform.
We hired 233 people in 2014 and ended the year with 575 employees. About 25% of our employees work from home. The other 75% work from offices around the world; Burlington MA (US), Portland OR (US), Washington DC (US), Paris (France), Reading (United Kingdom), Ghent (Belgium), Singapore, Delhi (India), Brisbane (Australia) and Sydney (Australia). About 75% of our employees are based in the United States. Despite our fast growth rate in staff, recruiting remains a key challenge; it's hard to hire as fast as we do and maintain the high bar we've set for ourselves in terms of talent and commitment.
We raised venture funding twice in 2014: a $50MM series F round led by New Enterprise Associates (NEA) followed by Amazon investing an undisclosed amount of money in our business. It's not like Tom Erickson and I enjoy raising money, but building and expanding a sales and marketing team is notoriously difficult and requires big investments. At the same time, we're building and supporting the development of multiple products in parallel. Most companies only build one product. We're going after a big dream to become the preferred platform for what has been called the "pivot point of many enterprise tech stacks" -- the technologies that permit organizations to deliver on the promises of exceptional digital customer experiences from an agile, open, resilient platform. We are also competing against behemoths. We can't show up to a gunfight with a knife, so to speak.
Building a digital platform for the enterprise
Digital has changed everything, and more and more organizations need or want to transform into digital-first businesses to stay in step with the preferences of their customers. Furthermore, technology innovations keep occurring at an ever faster and more disruptive pace. No organization is immune to the forces of digital disruption. At Acquia, we help our customers by providing a complete technology platform and the support necessary to support their digital initiatives. The Acquia Platform consists of tools and support for building and managing dynamic digital experiences. It includes Acquia Cloud, which helps developers deliver complex applications at scale, and Acquia Lift, our digital engagement services for bringing greater context to highly personalized experiences. Let me give you an update on each of the major components.
Drupal tools and support
Drupal gives organizations the ability to deliver a unified digital experience that includes mobile delivery, social and commerce. Great inefficiencies exist in most organizations that use a variety of different, disconnected systems to achieve those three essentials. They are tired of having to tie things together; content is important, social is important, commerce is important but connecting all these systems seamlessly and integrating them with preferred applications and legacy systems leads to massive inefficiencies. Companies want to do things well, and more often than not, Drupal allows them to do it better, more nimbly and in a far more integrated framework.
In 2010, we laid out our product vision and predicted more and more organizations would start to standardize on Drupal. Running 20 different content management systems on 20 different technology stacks is both an expensive and unnecessary burden. We've seen more and more large organizations re-platform most of their sites to Drupal and the Acquia Platform. They realize they don't need multiple content management systems for different sites. Great examples are Warner Music and Interscope Records, who have hundreds of sites on Drupal across the organization, resulting in significant cost savings and efficiency improvements. The success of our Acquia Cloud Site Factory solution has been gratifying to witness. According to a research study by Forrester Consulting, which we released late last year, ACSF is delivering a 944% return on investment to its adopters.
After many years of discussion and debate in the Drupal community, we launched the Acquia Certification Program in March 2014. So far, 546 Drupal developers from more than 45 countries have earned certification. The exams focus on real world experience, and the predominant comments we've heard this past year are that the exams are tough but fair. Acquia delivered six times the amount of training in 2014 compared to the previous year, and demand shows no sign of slowing.
Last, but definitely not least, is Drupal 8. We contributed significantly to Drupal 8 and helped it to achieve beta status; of the 513 critical Drupal 8 bugs fixed in 2014, Acquia's Office of the CTO helped fix 282 of them. We also funded work on the Drupal Module Upgrader to automate much of the work required to port modules from Drupal 7 to Drupal 8.
Drupal alone isn't enough for organizations to succeed in this digital-first world. In addition to adopting Drupal, the cloud continues to enable organizations to save time and money on infrastructure management so they can focus on managing websites more efficiently and bringing them to market faster. Acquia customers such as GRAMMY.com have come to depend on the Acquia Cloud to provide them with the kind of rugged, secure scale that ensures when the world's attention is focused on their sites, they will thrive. On a monthly basis, we're now serving more than 33 billion hits, almost 5 billion pageviews, 9 petabytes of data transferred, and logging 13 billion Drupal watchdog log lines. We added many new features to Acquia Cloud in 2014, including log streaming, self-service diagnosis tools, support for teams and permissions, two-factor authentication, new dashboards, improved security with support for Virtual Private Networks (VPNs), an API for Acquia Cloud, and more.
As powerful as the Drupal/Acquia Cloud combination may be, our customers demand far more from their digital properties, focusing more and more on optimizing them to fully deliver the best possible experience to each individual user. Great digital experiences have always been personal; today they have to become contextual, intuitively knowing each user and dynamically responding to each user's personal preference from device to location to history with the organization. After two years of development and the acquisition of TruCentric, we launched Acquia Lift in 2014.
It's surprising how many organizations aren't implementing any form of personalization today. Even the most basic level of user segmentation and targeting allows organizations to better serve their visitors and can translate into significant growth and competitive differentiation. Advanced organizations have a single, well-integrated view of the customer to optimize both the experience and the lifetime value of that customer, in a consistent fashion across all of their digital touchpoints. Personalization not only leads to better business results, customers have come to expect it and if they don't find it, they'll go elsewhere to get it. Acquia Lift enables organizations to leverage data from multiple sources in order to serve people with relevant content and commerce based on intent, locations and interests. I believe that Acquia Lift has tremendous opportunity and that it will grow to be a significant business in and of itself.
While our key areas of investment in 2014 were Acquia Cloud and Acquia Lift, we did a lot more. Our Mollom service blocked more than 7.8 billion spam messages with an error rate of only 0.01%. We continue to invest in commerce; we helped launch the new Puma website leveraging our Demandware connector and continue to invest and focus on the integration of content and commerce. Overall, the design and user experience of our products has improved a lot, but it is still an area for us to work on. Expect us to focus more heavily on user experience in 2015.
The results of all our efforts around the launch of the Acquia Platform have not gone unnoticed. In October, Acquia was identified as a Leader in the 2014 Gartner Magic Quadrant for Web Content Management.
The wind is blowing in the right direction
I'm very optimistic about Acquia's future in 2015. I believe we've steered the company to be positioned at the right place at the right time. As more organizations are shifting to becoming digital-first businesses they want to build digital experiences that are more pervasive, more contextual, more targeted, more integrated, and last but not least, more secure.
The consolidation from many individual point solutions to one platform is gaining momentum, although re-platforming is usually a long process. Organizations want the unified or integrated experience that Drupal has to offer, as well as the flexibility of Open Source. It is still time consuming and challenging to create quality content, and I believe there is plenty of opportunity for us and our partners to help with that going forward.
Without a doubt, organizations want to better understand their customers and use data-driven decisions to drive growth. Data is becoming the new product. The opportunity this creates in commerce is massive.
Cloud computing and Software-as-a-Service (SaaS) continues to be on the rise. Cloud is top of mind and the transition away from on-premise solutions is accelerating even as the arguments around security and privacy issues in the cloud continue to be raised. While there is a certain amount of emotion, and sometimes politics, people are beginning to realize that the cloud is usually more secure and more robust against cyber-attacks than traditional on-premise systems.
The promise of Drupal 8, arguably the most significant advance in the evolution of the Drupal software, has me very excited. It is shaping up to be a great release, and I'm confident it will further secure Drupal's reputation among developers, designers, agencies and site managers as the most flexible, powerful content management solution available.
All of this is not to say 2015 will be easy. This is an incredibly exciting and fast-changing space in the world of technology. Acquia is growing in an incredibly fast-paced, dynamic sector and we realize our mission is to help our customers understand how to think ahead to ever more innovation and change. Simplifying our overall messaging and defining ourselves around the Acquia Platform is a significant first step.
Of course, none of this success would be possible without the support of our customers, partners, the Drupal community, the Acquia team, and our many friends. Thank you for your support in 2014, and I look forward to working with you to find out what 2015 will bring!
The older I get, the quicker the years seem to fly by. As I begin to reflect on a great 2014, one thing is crystal clear again. People are the most important thing to any organization. Having a great team is more important than having a great idea. A good team will figure out how to make something great happen; they'll pivot, evolve and claw their way to success. I see it every day at Acquia, the Drupal Association or the Drupal community. I'm fortunate to be surrounded by so many great people.
By extension, recruiting is serious business. How do you figure out if someone is a great fit for your organization? Books have been written about finding and attracting the right people, but for me the following quote from Dee Hock, the founder of Visa, sums it up perfectly.
"Hire and promote first on the basis of integrity; second, motivation; third, capacity; fourth, understanding; fifth, knowledge; and last and least, experience. Without integrity, motivation is dangerous; without motivation, capacity is impotent; without capacity, understanding is limited; without understanding, knowledge is meaningless; without knowledge, experience is blind." — Dee Hock, founder of Visa.
Most hiring managers get it wrong and focus primarily on experience. While experience can be important, attitude is much more important. Attitude, not experience, is what creates a strong positive culture and what turns users and customers into raving fans.