Turning Drupal outside-in

There has been a lot of discussion around the future of the Drupal front end both on Drupal.org (#2645250, #2645666, #2651660, #2655556) and on my blog posts about the future of decoupled Drupal, why a standard framework in core is a good idea, and the process of evaluating frameworks. These all relate to my concept of "progressive decoupling", in which some portions of the page are handed over to client-side logic after Drupal renders the initial page (not to be confused with "full decoupling").

My blog posts have drawn a variety of reactions. Members of the Drupal community, including Lewis Nyman, Théodore Biadala and Campbell Vertesi, have written blog posts with their opinions, as well as Ed Faulkner of the Ember community. Last but not least, in response to my last blog post, Google changed Angular 2's license from Apache to MIT for better compatibility with Drupal. I read all the posts and comments with great interest and wanted to thank everyone for all the feedback; the open discussion around this is nothing short of amazing. This is exactly what I hoped for: community members from around the world brainstorming about the proposal based on their experience, because only with the combined constructive criticism will we arrive at the best solution possible.

Based on the discussion, rather than selecting a client-side JavaScript framework for progressive decoupling right now, I believe the overarching question the community wants to answer first is: How do we keep Drupal relevant and widen Drupal's adoption by improving the user experience (UX)?

Improving Drupal's user experience is a topic near and dear to my heart. Drupal's user experience challenges led to my invitation to Mark Boulton to redesign Drupal 7, the creation of the Spark initiative to improve the authoring experience for Drupal 8, and continued support for usability-related initiatives. In fact, the impetus behind progressive decoupling and adopting a client-side framework is the need to improve Drupal's user experience.

It took me a bit longer than planned, but I wanted to take the time to address some of the concerns and share more of my thoughts about improving Drupal's UX (and JavaScript frameworks).

To iterate or to disrupt?

In his post, Lewis writes that the issues facing Drupal's UX "go far deeper than code" and that many of the biggest problems found during the Drupal 8 usability study last year are not resolved with a JavaScript framework. This is true; the results of the Drupal 8 usability study show that Drupal can confuse users with its complex mental models and terminology, but it also shows how modern features like real-time previews and in-page block insertion are increasingly assumed to be available.

To date, much of our UX improvements have been based on an iterative process, meaning it converges on a more refined end state by removing problems in the current state. However, we also require disruptive thinking, which is about introducing entirely new ideas, for true innovation to happen. It's essentially removing all constraints and imagining what an ideal result would look like.

I think we need to recognize that while some of the documented usability problems coming out of the Drupal 8 usability study can be addressed by making incremental changes to Drupal's user experience (e.g. our terminology), other well-known usability problems most likely require a more disruptive approach (e.g. our complex mental model). I also believe that we must acknowledge that disruptive improvements are possibly more impactful in keeping Drupal relevant and widening Drupal's adoption.

At this point, to get ahead and lead, I believe we have to do both. We have to iterate and disrupt.

From inside-out to outside-in

Let's forget about Drupal for a second and observe the world around us. Think of all the web applications you use on a regular basis, and consider the interaction patterns you find in them. In popular applications like Slack, the user can perform any number of operations to edit preferences (such as color scheme) and modify content (such as in-place editing) without incurring a single full page refresh. Many elements of the page can be changed without the user's flow being interrupted. Another example is Trello, in which users can create new lists on the fly and then add cards to them without ever having to wait for a server response.

Contrast this with Drupal's approach, where any complex operation requires the user to have detailed prior knowledge about the system. In our current mental model, everything begins in the administration layer at the most granular level and requires an unmapped process of bottom-up assembly. A user has to make a content type, add fields, create some content, configure a view mode, build a view, and possibly make the view the front page. If each individual step is already this involved, consider how much more difficult it becomes to traverse them in the right order to finally see an end result. While very powerful, the problem is that Drupal's current model is "inside-out". This is why it would be disruptive to move Drupal towards an "outside-in" mental model. In this model, I should be able to start entering content, click anything on the page, seamlessly edit any aspect of its configuration in-place, and see the change take effect immediately.

Drupal 8's in-place editing feature is actually a good start at this; it enables the user to edit what they see without an interrupted workflow, with faster previews and without needing to find what thing it is before they can start editing.

Making it real with content modeling

Eight years ago in 2007, I wrote about a database product called DabbleDB. I shared my belief that it was important to move CCK and Views into Drupal's core and learn from DabbleDB's integrated approach. DabbleDB was acquired by Twitter in 2010 but you can still find an eight-year-old demo video on YouTube. While the focus of DabbleDB is different, and the UX is obsolete, there is still a lot we can learn from it today: (1) it shows a more integrated experience between content creation, content modeling, and creating views of content, (2) it takes more of an outside-in approach, (3) it uses a lot less intimidating terminology while offering very powerful capabilities, and (4) it uses a lot of in-place editing. At a minimum, DabbleDB could give us some inspiration for what a better, integrated content modeling experience could look like, with the caveat that the UX should be as effortless as possible to match modern standards.

Other new data modeling approaches with compelling user experiences have recently entered the landscape. These include back end-as-a-service (BEaaS) solutions such as Backand, which provides a visually clean drag-and-drop interface for data modeling and helpful features for JavaScript application developers. Our use cases are not quite the same, but Drupal would benefit immensely from a holistic experience for content modeling and content views that incorporates both the rich feature set of DabbleDB and the intuitive UX of Backand.

This sort of vision was not possible in 2007 when CCK was a contributed module for Drupal 6. It still wasn't possible in Drupal 7 when Views existed as a separate contributed module. But now that both CCK and Views are in Drupal 8 core, we can finally start to think about how we can more deeply integrate the two. This kind of integration would be nontrivial but could dramatically simplify Drupal's UX. This should be really exciting because so many people are attracted to Drupal exactly because of features like CCK and Views. Taking an integrated approach like DabbleDB, paired with a seamless and easy-to-use experience like Slack, Trello and Backand, is exactly the kind of disruptive thinking we should do.

What most of the examples above have in common are in-place editing, immediate previews, no page refreshes, and non-blocking workflows. The implications on our form and render systems of providing configuration changes directly on the rendered page are significant. To achieve this requires us to have robust state management and rendering on the client side as well as the server side. In my vision, Twig will provide structure for the overall page and non-interactive portions, but more JavaScript will more than likely be necessary for certain parts of the page in order to achieve the UX that all users of the web have come to expect.

We shouldn't limit ourselves to this one example, as there are a multitude of Drupal interfaces that could all benefit from both big and small changes. We all want to improve Drupal's user experience — and we have to. To do so, we have to constantly iterate and disrupt. I hope we can all collaborate on figuring out what that looks like.

Special thanks to Preston So and Kevin O'Leary for contributions to this blog post and to Wim Leers for feedback.

Dries meets the King and Queen of Belgium

A few weeks ago at Davos, I had the honor of meeting the King and Queen of Belgium. VTM, the main commercial television station in Belgium, captured the moment for the evening news. Proof in the video below!

Fast tracking life-saving innovations

Last week, a member of my family died in a car accident. Jasper was on his way home and was hit by a taxi. He fought for his life, but died the next day. Jasper was only 16 years old. I was at Davos and at one point I had to step out of the conference to cry. Five years ago, another family member died after she was hit by a truck when crossing the road.

It's hard to see a tragedy like this juxtaposed against a conference filled with people talking about improving the state of the world. These personal losses make me want to fast-forward to a time in the future where self-driving cars are normal, and life-saving innovations don't have as much regulatory red tape to cut through before they can have an impact. It's frustrating that we may have the right technology in sight today, but aren't making it available, especially when people's lives are at stake.

Imagine two busses full of people crashing, killing everyone on board, every single day. That is how many people die on America's roads every day. In fact, more people are killed by cars than guns, but I don't see anyone calling for a ban on automobiles. Car accidents (and traffic jams) are almost always the result of human error. It is estimated that self-driving cars could reduce deaths on the road by 90%. That is almost 30,000 lives saved each year in the US alone. The life-saving estimates for driverless cars are on par with the efficacy of modern vaccines. I hope my children, now ages 6 and 8, will never need a driver's license and can grow up in a world with driverless cars.

The self-driving car isn't as far off as you might think but is still being held back by government regulators. Delayed technology isn't limited to self-driving cars. Life-saving innovations in healthcare are often held back by regulatory requirements. The challenge of climate change could be addressed faster if the regulatory uncertainty around solar and wind power permits and policies were reduced. The self-serving interest of lobbying groups focused on maintaining the status quo for industries like Big Oil make it harder for alternative energies to gain momentum.

Regulators need to frame their jobs differently; they need to ask how they can facilitate and enable emerging disruptive innovations, rather than maintain existing systems. Their job should focus more on removing any barriers that prevent disruptions from having a faster impact. If they do this job well, some established institutions will fail. In some cases, economic sacrifices by the incumbents should be of lesser concern than advancing social health and safety for the benefit of society. I'm less concerned about technology destroying jobs, and more concerned about our children not being able to benefit from available technical advances that improve their lives. We should realize that opportunities for long-term economic growth come with short-term disruption or temporary pain.

Losing family members in fatal accidents makes one think about what could have been done. I'm often asked how one can create a "Silicon Valley" model elsewhere in the world. I may have an answer. If you want to out-"Silicon Valley" Silicon Valley, create a region with a regulatory environment that supports prompt, responsible innovation to drive the adoption and iteration of new technologies. A region where people can responsibly launch self-driving cars, fast-track healthcare and address climate change. A region where long-term advantages are valued more than short-term disadvantages. Such a region would attract capital and entrepreneurs, and would be much better for our children.

Drupal: 15 years old and still gaining momentum

On December 29, 2000, I made a code commit that would change my life; it is in this commit that I called my project "Drupal" and added the GPL license to it.

Drupal name and license

The commit where I dubbed my website project "Drupal" and added the GPL license.

A couple weeks later, on January 15, 2001, exactly 15 years ago from today, I released Drupal 1.0.0 into the world. The early decisions to open-source Drupal and use the GPL license set the cornerstone principles for how our community shares with one another and builds upon each other's achievements to this day.

Drupal is now 15 years old. In internet terms, that is an eternity. In 2001, only 7 percent of the world's population had internet access. The mobile internet had not entered the picture, less than 50% of the people in the United States had a mobile phone, and AT&T had just introduced text messaging. People searched the web with Lycos, Infoseek, AltaVista and Hot Bot. Google -- launched in 1998 as a Stanford University research project -- was still a small, private company just beginning its rise to prominence. Google AdWords, now a $65 billion business, had less than 500 customers when Drupal launched. Chrome, Firefox, and Safari didn't exist yet; most people used Netscape, Opera or Internet Explorer. New ideas for sharing and exchanging content such as "public diaries" and RSS had yet to gain widespread acceptance and Drupal was among the first to support those. Wikipedia was launched on the same day as Drupal and sparked the rise of user-generated content. Facebook and Twitter didn't exist until 4-5 years later. Proprietary software vendors started to feel threatened by open source; most didn't understand how a world-class operating system could coalesce out of part-time hacking by several thousand developers around the world.

Looking back, Drupal has not only survived massive changes in our industry; it has also helped drive them. Over the past decade and a half, I've seen many content management systems emerge and become obsolete: Vignette, Interwoven, PHP-Nuke, and Scoop were all popular at some point in the past but Drupal has outlived them all. A big reason is from the very beginning we have been about constant evolution and reinvention, painful as it is.

Keeping up with the pace of the web is a funny thing. Sometimes you'll look back on choices made years ago and think, "Well, I'm glad that was the right decision!". For example, Drupal introduced "hooks" and "modules" early on, concepts that are commonplace in today's platforms. At some point, you could even find some of my code in WordPress, which Matt Mullenweg started in 2003 with some inspiration from Drupal. Another fortuitous early decision was to focus Drupal on the concept of "nodes" rather than "pages". It wasn't until 10 years later with the rise of mobile that we started to see the web revolve less and less around pages. A node-based approach makes it possible to reuse content in different ways for different devices. In a way, much of the industry is still catching up to that vision. Even though the web is a living, breathing thing, there is a lot of things that we got right.

Other times, we got it wrong. For example, we added support for OpenID, which never took off. In the early days I focused, completely and utterly, on the aesthetics of Drupal's code. I spent days trying to do something better, with fewer lines of code and more elegant than elsewhere. But in the process, I didn't focus enough on end-user usability, shunned JavaScript for too long, and later tried to improve usability by adding a "dashboard" and "overlay".

In the end, I feel fortunate that our community is willing to experiment and break things to stay relevant. Most recently, with the release of Drupal 8, we've made many big changes that will fuel Drupal's continued adoption. I believe we got a lot of things right in Drupal 8 and that we are on the brink of another new and bright era for Drupal.

I've undergone a lot of personal reinvention over the past 15 years too. In the early days, I spent all my time writing code and building Drupal.org. I quickly learned that a successful open source project requires much more than writing code. As Drupal started to grow, I found myself an "accidental leader" and worried about our culture, scaling the project, attracting a strong team of contributors, focusing more and more on Drupal's end-users, growing the commercial ecosystem around Drupal, starting the Drupal Association, and providing vision. Today, I wear a lot of different hats: manager of people and projects, evangelist, fundraiser, sponsor, public speaker, and BDFL. At times, it is difficult and overwhelming, but I would not want it any other way. I want to continue to push Drupal to reach new heights and new goals.

Today we risk losing much of the privacy, serendipity and freedom of the web we know. As the web evolves from a luxury to a basic human right, it's important that we treat it that way. To increase our impact, we have to continue to make Drupal easier to use. I'd love to help build a world where people's privacy is safe and Drupal is more approachable. And as the pace of innovation continues to accelerate, we have to think even more about how to scale the project, remain agile and encourage experimentation. I think about these issues a lot, and am fortunate enough to work with some of the smartest people I know to build the best possible version of the web.

So, here is to another 15 years of evolution, reinvention, and continued growth. No one knows what the web will look like 15 years in the future, but we'll keep doing our best to guide Drupal responsibly.

Selecting a client-side framework for Drupal

Last month, my blog post about whether a client-side framework is right for Drupal stimulated some excellent insights into how the future of the Drupal front end might look. There was broad agreement that incorporating more JavaScript into Drupal's administrative interface is important for a future-ready user experience.

I am confident that adopting a client-side framework through progressive decoupling will give us the best of both worlds. Of course, this does not mean I oppose fully decoupling through any other framework; in fact, I believe we should redouble our efforts toward a first-class API-first Drupal. But progressive decoupling means that we will be able to work toward a next-generation user experience without abandoning much of the work we've done so far.

With that in mind, I tasked a small team made up of various experts from the Drupal and various JavaScript communities with putting together a list of criteria and a comparison matrix to help us decide on the most appropriate client-side framework for progressive decoupling in Drupal.

JavaScript framework comparison matrix

After a hundred hours of work, we came up with a criteria document and comparison matrix (PDF version):

Special thanks to all of the following experts who provided review and input: Miško Hevery (creator of Angular; Google) and Igor Minar (technical lead for Angular; Google); Ed Faulkner (core maintainer for Ember); Amitai Burstein (Drupal and Elm contributor; Gizra); Sebastian Siemssen (Drupal contributor, Elm and React developer; Zensations); and John Albin Wilkins (Drupal 8 mobile initiative lead), Alex Bronstein (Drupal core maintainer; Acquia), Wim Leers (Drupal core contributor; Acquia), and Preston So (Drupal contributor, Acquia).

Though this is a good starting point that lays the groundwork for a formal adoption process in Drupal core, it is time to open our comparison for review by members of both the Drupal and JavaScript communities. We should also more rigorously evaluate these frameworks' appropriateness through actual development.

First, have we decided on the right criteria regardless of the frameworks themselves? This is probably the most important at this stage. While many organizations choose to adopt client-side frameworks for fully decoupled implementations, Drupal is the first to consider layering a framework on top to allow both richly dynamic and more traditional modules to coexist gracefully through progressive decoupling. What issues around templates, rendering, and client-side caching should we incorporate into these criteria? Is there anything missing or overemphasized?

Second, have we selected the right frameworks to focus on? Are there other frameworks, libraries, or even compile-to-JavaScript projects (such as Elm) that should be included in the matrix? In my view, it is best for Drupal to adopt a framework with an already large community and ecosystem that would allow us to more quickly bridge the gap and resolve any friction during adoption. To provide a good frame of reference, we've also included Backbone, Angular, and Knockout, all three slightly older among client-side frameworks. Others on our shortlist that we didn't consider due to low levels of adoption and small communities are Mithril, Riot, and Vue. Still other ambitious projects such as the Elm and ClojureScript languages are also candidates, but their adoption will mean more up-front work to support typical features of client-side frameworks, as well as buy-in into an approach where JavaScript is compiled from a different language.

Finally, have we drawn the right conclusions against these criteria? In other words, did we fill out the cells correctly? While they have been reviewed by some of the frameworks' experts, there might be unexpected gotchas or caveats.

I'm sharing the initial comparison matrix on my blog for maximum reach; I want both the Drupal community and the JavaScript framework communities, as well as the broader front-end community, to take note. After three installments on my blog ("The future of decoupled Drupal", "Should we decouple Drupal with a client-side framework?"), it's time to move the technical conversation to drupal.org. There is now an issue in the core issue queue on drupal.org to iterate on the matrix.

Preliminary conclusions

At the moment, the most promising candidates in the comparison matrix appear to be Angular 2, Ember, and React, given their technical robustness, relative suitability for progressively decoupled Drupal, and their strong levels of community support and broader adoption. Given that Backbone is already in core and several modules already rely on it, we have included it too.

What we've learned from talking to the different projects is that they are often converging on similar techniques and best practices; they are by and large adding support for Virtual DOM implementations or rehydration (seamless state transfer), and they are all obsessing over small payload size and performance, better testability, etc. Therefore it is important to focus on the fundamental, often philosophical, differences between the projects that will likely be unchanged in time; key architectural differences, their release cadence and stance on backward compatibility, their license, their governance model, their flexibility and learning curve, etc.

From a quick glance at the criteria and our needs, it seems that Ember is currently our best candidate, as it appears to have a slight technical edge overall. Ember 2.0 has an all-new rendering engine named Glimmer, and it has server-side rendering through FastBoot. On the other hand, however, Ember is quite bulky and opinionated (enforcing patterns for code structure) compared to other candidate frameworks. A more fundamental difference is that unlike Angular and React, which have corporate governance and funding, Ember is a community-driven project like Drupal.

While React is lightweight, it needs integration with a variety of other libraries in the React ecosystem to work as a full-fledged implementation, which gives it a steep learning curve from an implementation standpoint. Because React is a relatively young project, best practices are shifting quickly and making it less attractive. The Virtual DOM, among React's most compelling features, has also seen its core ideas filter into other framework projects. But more importantly, React is licensed with what I believe to be a potentially unacceptable patent clause, which states that an organization can no longer use React once it sues Facebook for any (unrelated) patent infringement. This has already generated some concerted pushback from both WordPress's Calypso and React contributors.

Angular 2 is a complete rewrite of Angular 1 to address issues with that version of the framework, including performance problems stemming from a large file size. The new version also introduces a new template engine that incorporates both directives familiar to Angular 1 developers and nestable component-based templates. But Angular 2's Apache 2.0 licensing is incompatible with Drupal's own GPLv2 license. While Drupal's PHP code and JavaScript code run in isolated processes, it appears that an Apache 2.0-licensed project can't be jointly distributed within an umbrella project that uses a GPLv2 license.

In addition to being at a slight technical disadvantage to Ember, the legal concerns with React and Angular make me believe Ember might be our best bet. But I'm not a lawyer nor a JavaScript expert, so I can't make this decision alone; I'm looking for lots of feedback, particularly from JavaScript experts, to determine the best option for Drupal.

Conclusion

Whatever the result of the debate around which client-side framework to adopt, I believe that Drupal needs to move quickly to embrace a framework that will aid development of a progressively decoupled architecture and corresponding user experience improvements. By providing some baseline criteria and including our accomplished community, I have no doubt we can reach this decision quickly but also intelligently.

Special thanks to Preston So for contributions to this blog post.

Pages

Dries Buytaert is the original creator and project lead of Drupal and the co-founder and CTO of Acquia. He writes about Drupal, startups, business, photography and building the world we want to exist in.

Updates from Dries straight to your mailbox