When traffic skyrockets your site shouldn't go down

This week's Grammy Awards is one of the best examples of the high traffic events websites that Acquia is so well known for. This marks the fourth time we hosted the Grammys' website. We saw close to 5 million unique visitors requesting nearly 20 million pages on the day of the awards and the day after. From television's Emmys to Superbowl advertisers' sites, Acquia has earned its reputation for keeping their Drupal sites humming during the most crushing peaks of traffic.

These "super spikes" aren't always fun. For the developers building these sites to the producers updating each site during the event, nothing compares to the sinking feeling when a site fails when it is needed the most. During the recent Superbowl, one half-time performer lost her website (not on Drupal), giving fans the dreaded 503 Service Unavailable error message. According to CMSWire: "Her website was down well past midnight for those who wanted to try and score tickets for her tour, announced just after her halftime show performance". Yet for Bruno Mars' fans, his Acquia-based Drupal site kept rolling even as millions flooded his site during the half-time performance.

For the Grammys, we can plan ahead and expand their infrastructure prior to the event. This is easy thanks to Acquia Cloud's elastic platform capacity. Our technical account managers and support teams work with the producers at the Grammys to make sure the right infrastructure and configuration is in place. Specifically, we simulate award night traffic as best we can, and use load testing to prepare the infrastructure accordingly. If needed, we add additional server capacity during the event itself. Just prior to the event, Acquia takes a 360 degree look at the site to ensure that all of the stakeholders are aligned, whether internal to Acquia or external at a partner. We have technical staff on site during the event, and remote teams that provide around the clock coverage before and after the event.

Few people know what goes on behind the scenes during these super spikes, but the biggest source of pride is that our work is often invisible; our job well done means that our customer's best day, didn't turn into their worst day.

In memoriam: Richard Burford

It is with great sadness that we learned last week that Richard Burford has passed away. This is a tragic loss for his family, for Acquia, the Drupal community, and the broader open source world. Richard was a Sr. Software Engineer at Acquia for three and a half years (I still remember him interviewing with me), and known as psynaptic in the Drupal community. Richard has been a member of the Drupal community for 9+ years. During that time, he contributed hundreds of patches across multiple projects, started a Drupal user group in his area and helped drive the Drupal community in the UK where he lived. Richard was a great person, a dedicated and hard-working colleague, a generous contributor to Drupal, and a friend. Richard was 36 years young with a wife and 3 children. He was the sole income earner for the family so a fundraising campaign has been started to help out his family during these difficult times; please consider contributing.

Should we prioritize technological advances?

This blog post is co-authored with Devesh Raj, Senior Vice President and Head of Strategy and Planning at Comcast-NBCU. Devesh and I are friends and both Young Global Leader at the World Economic Forum. In this blog post we share some of our observations and thoughts after attending the World Economic Forum's annual meeting in Davos.

This year's World Economic Forum Annual Meeting in Davos focused on the Fourth Industrial Revolution, a term coined by Klaus Schwab to describe the new generation of technological advances – sensors, robotics, artificial intelligence, 3D printing, precision medicine – coming together to define the next wave of progress.

These new technologies have the potential to transform our lives. Beyond sci-fi like scenarios – such as each of us having our own personal R2-D2, summoning our Batmobile, or colonizing Mars – these advances also have the potential to solve many real-world problems. With more intelligent, automated technology, we could generate renewable energy, address climate change, connect billions of people to the internet, develop affordable housing solutions and cure chronic diseases.

These advances are not far into the future. A recent report on Technology Tipping Points and Societal Impact anticipates many such moments of inflection within our lifetimes – in fact, we may see major advances in transportation, artificial intelligence, and new payment technology as soon as the next decade. Yet, somewhat surprisingly, much of the discussion in Davos last month focused on the negative impacts of these technologies, rather than their positive potential.

The average year that each tipping point is expected to occur. Source: WEF report.
2018 2021 2022 2023 2024 2025 2026
- Storage for all - Robots and services - The Internet of Things
- Wearable internet
- 3D printing and manufacturing
- Implantable technologies
- Big Data for decisions
- Vision as the new interface
- Our digital presence
- Governments and the block chain
- A supercomputer in your pocket
- Ubiquitous computing
- 3D printing and human health
- The connected home
- 3D printing and consumer products
- AI and white-collar jobs
- The sharing economy
- Driverless cars
- AI and decision-making
- Smart cities

One consistent, fearful theme was the potential for job losses. As automation continues to replace manufacturing or blue collar jobs, artificial intelligence will subsequently do the same for skilled, white collar jobs in banking, law or medicine. Estimates as to the impact this will have on jobs vary, but many prognostications in Davos suggested a depressive impact on the global economy. While it's true that technological leaps have often eliminated older, human-powered methods of doing things, many in Davos also recognized that advances in technology create new jobs, most of which we can't even dream of today. For example, the invention of the airplane created hundreds of thousands of jobs, from pilots, to stewards, to airport personnel, to international agents and more prognostications not to mention the transformative economic impact of billions of people traveling vast distances in a short span of time.

A second concern at Davos was growing inequality in the world between "digital haves" and "have-nots". This was reflected both as a challenge among nations – developed vs. developing – but also an issue for specific socio-economic groups within individual nations, some of which arguably are still not past the second or third industrial revolution. What does 3D printing or precision medicine do, for example, for rural parts of India and Africa that still don't have reliable electricity, while urban centers in those same countries race towards an era of smart, automated living?

A third common concern (particularly driven by robotics and artificial intelligence) was the "dehumanization" of our lives. There was a case for a renewed emphasis on qualities that make us uniquely human – empathy, sensitivity, creativity and inspiration.

Another issue centered on the ethical and moral challenges of many advances. Some conversations at Davos discussed the dangerous potential of eugenics-like scenarios in medicine, enabled by advances such as CRISPR/Cas9. On the flip side, could machines make positive decisions regarding human lives, such as a self-driving car making a choice between hitting a pedestrian or sacrificing its passenger?

One could argue some of these concerns are overblown Luddism. But in some ways, it doesn't matter – the march of technological progress is inevitable, as it has always been. Certainly, no one at Davos suggested slowing down the pace of technological advancement. The gist of the discussions was that we should figure out how to avoid, or address, the negative, unintended consequences of these changes.

We believe there is a major challenge with the Fourth Industrial Revolution that didn't get adequate attention in Davos – the issue of prioritization.

To date, the technological innovation that has driven the Fourth Industrial Revolution is shaped by the commercial prospects of small or large firms in the market. After all, one definition of "innovation" is the commercial application of invention. As an example, investment in alternative energy R&D fluctuates depending on oil prices, just as demand for hybrid or electric vehicles become more or less attractive depending on gasoline prices.

What if, instead of being driven solely by commercial returns, we could focus the Fourth Industrial Revolution more directly on the big problems our world faces? What if we could prioritize technological advances that have the most beneficial impact to society?

The world has recently defined its problems very clearly in a set of 17 Sustainable Development Goals (Wikipedia), also known as the Global Goals, that were adopted by all countries last year to "end poverty, protect the planet and ensure prosperity for all". The goals cover poverty, hunger and food security, health, education, energy, and water and sanitation – to name a few. A successor list to the earlier Millennium Development Goals, the Sustainable Development Goals get quite specific.

Take Goal 3 as an example: "Ensure healthy lives and promote well-being for all at all ages". This goal is linked to 12 targets, including these top three:

  • By 2030: reduce the global maternal mortality ratio to less than 70 per 100,000 live births.
  • By 2030: end preventable deaths of newborns and children under 5 years of age, with all countries aiming to reduce neonatal mortality to at least as low as 12 per 1,000 live births and under-5 mortality to at least as low as 25 per 1,000 live births.
  • By 2030: end the epidemics of AIDS, tuberculosis, malaria and neglected tropical diseases and combat hepatitis, water-borne diseases and other communicable diseases.

Of course, technological advancement is not the only solution to all Sustainable Development Goals – there is much more to do – but it is likely one of the major contributors.

As the world thinks through how to harness the Fourth Industrial Revolution, we think it is worth questioning which technologies we should be prioritizing to meet these Sustainable Development Goals. How do we draft policies and create economic incentives to encourage the right types of technology advances? What should governments and the private sector do differently to focus technology on addressing these goals? How do we direct the energy and creativity of millions of entrepreneurs towards improving the state of the world?

The world's innovation system is powerful and has generally worked well. However, it could use a guiding hand to nudge it in a direction that will benefit the planet beyond the incentives of commercial returns. Expanding our criteria for importance to solving areas of global need is not an inherently anti-capitalist idea. But it is one that would channel capitalism in the best direction for humanity as a whole. That, we hope, is the real agenda initiated by the focus in Davos on the Fourth Industrial Revolution, which the world will seek to address in the coming year.

Turning Drupal outside-in

There has been a lot of discussion around the future of the Drupal front end both on Drupal.org (#2645250, #2645666, #2651660, #2655556) and on my blog posts about the future of decoupled Drupal, why a standard framework in core is a good idea, and the process of evaluating frameworks. These all relate to my concept of "progressive decoupling", in which some portions of the page are handed over to client-side logic after Drupal renders the initial page (not to be confused with "full decoupling").

My blog posts have drawn a variety of reactions. Members of the Drupal community, including Lewis Nyman, Théodore Biadala and Campbell Vertesi, have written blog posts with their opinions, as well as Ed Faulkner of the Ember community. Last but not least, in response to my last blog post, Google changed Angular 2's license from Apache to MIT for better compatibility with Drupal. I read all the posts and comments with great interest and wanted to thank everyone for all the feedback; the open discussion around this is nothing short of amazing. This is exactly what I hoped for: community members from around the world brainstorming about the proposal based on their experience, because only with the combined constructive criticism will we arrive at the best solution possible.

Based on the discussion, rather than selecting a client-side JavaScript framework for progressive decoupling right now, I believe the overarching question the community wants to answer first is: How do we keep Drupal relevant and widen Drupal's adoption by improving the user experience (UX)?

Improving Drupal's user experience is a topic near and dear to my heart. Drupal's user experience challenges led to my invitation to Mark Boulton to redesign Drupal 7, the creation of the Spark initiative to improve the authoring experience for Drupal 8, and continued support for usability-related initiatives. In fact, the impetus behind progressive decoupling and adopting a client-side framework is the need to improve Drupal's user experience.

It took me a bit longer than planned, but I wanted to take the time to address some of the concerns and share more of my thoughts about improving Drupal's UX (and JavaScript frameworks).

To iterate or to disrupt?

In his post, Lewis writes that the issues facing Drupal's UX "go far deeper than code" and that many of the biggest problems found during the Drupal 8 usability study last year are not resolved with a JavaScript framework. This is true; the results of the Drupal 8 usability study show that Drupal can confuse users with its complex mental models and terminology, but it also shows how modern features like real-time previews and in-page block insertion are increasingly assumed to be available.

To date, much of our UX improvements have been based on an iterative process, meaning it converges on a more refined end state by removing problems in the current state. However, we also require disruptive thinking, which is about introducing entirely new ideas, for true innovation to happen. It's essentially removing all constraints and imagining what an ideal result would look like.

I think we need to recognize that while some of the documented usability problems coming out of the Drupal 8 usability study can be addressed by making incremental changes to Drupal's user experience (e.g. our terminology), other well-known usability problems most likely require a more disruptive approach (e.g. our complex mental model). I also believe that we must acknowledge that disruptive improvements are possibly more impactful in keeping Drupal relevant and widening Drupal's adoption.

At this point, to get ahead and lead, I believe we have to do both. We have to iterate and disrupt.

From inside-out to outside-in

Let's forget about Drupal for a second and observe the world around us. Think of all the web applications you use on a regular basis, and consider the interaction patterns you find in them. In popular applications like Slack, the user can perform any number of operations to edit preferences (such as color scheme) and modify content (such as in-place editing) without incurring a single full page refresh. Many elements of the page can be changed without the user's flow being interrupted. Another example is Trello, in which users can create new lists on the fly and then add cards to them without ever having to wait for a server response.

Contrast this with Drupal's approach, where any complex operation requires the user to have detailed prior knowledge about the system. In our current mental model, everything begins in the administration layer at the most granular level and requires an unmapped process of bottom-up assembly. A user has to make a content type, add fields, create some content, configure a view mode, build a view, and possibly make the view the front page. If each individual step is already this involved, consider how much more difficult it becomes to traverse them in the right order to finally see an end result. While very powerful, the problem is that Drupal's current model is "inside-out". This is why it would be disruptive to move Drupal towards an "outside-in" mental model. In this model, I should be able to start entering content, click anything on the page, seamlessly edit any aspect of its configuration in-place, and see the change take effect immediately.

Drupal 8's in-place editing feature is actually a good start at this; it enables the user to edit what they see without an interrupted workflow, with faster previews and without needing to find what thing it is before they can start editing.

Making it real with content modeling

Eight years ago in 2007, I wrote about a database product called DabbleDB. I shared my belief that it was important to move CCK and Views into Drupal's core and learn from DabbleDB's integrated approach. DabbleDB was acquired by Twitter in 2010 but you can still find an eight-year-old demo video on YouTube. While the focus of DabbleDB is different, and the UX is obsolete, there is still a lot we can learn from it today: (1) it shows a more integrated experience between content creation, content modeling, and creating views of content, (2) it takes more of an outside-in approach, (3) it uses a lot less intimidating terminology while offering very powerful capabilities, and (4) it uses a lot of in-place editing. At a minimum, DabbleDB could give us some inspiration for what a better, integrated content modeling experience could look like, with the caveat that the UX should be as effortless as possible to match modern standards.

Other new data modeling approaches with compelling user experiences have recently entered the landscape. These include back end-as-a-service (BEaaS) solutions such as Backand, which provides a visually clean drag-and-drop interface for data modeling and helpful features for JavaScript application developers. Our use cases are not quite the same, but Drupal would benefit immensely from a holistic experience for content modeling and content views that incorporates both the rich feature set of DabbleDB and the intuitive UX of Backand.

This sort of vision was not possible in 2007 when CCK was a contributed module for Drupal 6. It still wasn't possible in Drupal 7 when Views existed as a separate contributed module. But now that both CCK and Views are in Drupal 8 core, we can finally start to think about how we can more deeply integrate the two. This kind of integration would be nontrivial but could dramatically simplify Drupal's UX. This should be really exciting because so many people are attracted to Drupal exactly because of features like CCK and Views. Taking an integrated approach like DabbleDB, paired with a seamless and easy-to-use experience like Slack, Trello and Backand, is exactly the kind of disruptive thinking we should do.

What most of the examples above have in common are in-place editing, immediate previews, no page refreshes, and non-blocking workflows. The implications on our form and render systems of providing configuration changes directly on the rendered page are significant. To achieve this requires us to have robust state management and rendering on the client side as well as the server side. In my vision, Twig will provide structure for the overall page and non-interactive portions, but more JavaScript will more than likely be necessary for certain parts of the page in order to achieve the UX that all users of the web have come to expect.

We shouldn't limit ourselves to this one example, as there are a multitude of Drupal interfaces that could all benefit from both big and small changes. We all want to improve Drupal's user experience — and we have to. To do so, we have to constantly iterate and disrupt. I hope we can all collaborate on figuring out what that looks like.

Special thanks to Preston So and Kevin O'Leary for contributions to this blog post and to Wim Leers for feedback.

Dries meets the King and Queen of Belgium

A few weeks ago at Davos, I had the honor of meeting the King and Queen of Belgium. VTM, the main commercial television station in Belgium, captured the moment for the evening news. Proof in the video below!


Dries Buytaert is the original creator and project lead of Drupal and the co-founder and CTO of Acquia. He writes about Drupal, startups, business, photography and building the world we want to exist in.

Updates from Dries straight to your mailbox