You are here

Improving Drupal's content workflow

At DrupalCon Mumbai I sat down for several hours with the Drupal team at Pfizer to understand the work they have been doing on improving Drupal content management features. They built a set of foundational modules that help advance Drupal's content workflow capabilities; from content staging, to multi-site content staging, to better auditability, offline support, and several key user experience improvements like full-site preview, and more. In this post, I want to point a spotlight on some of Pfizer's modules, and kick-off an initial discussion around the usefulness of including some of these features in core.

Use cases

Before jumping to the technical details, let's talk a bit more about the problems these modules are solving.

  1. Cross-site content staging — In this case you want to synchronize content from one site to another. The first site may be a staging site where content editors make changes. The second site may be the live production site. Changes are previewed on the stage site and then pushed to the production site. More complex workflows could involve multiple staging environments like multiple sites publishing into a single master site.
  2. Content branching — For a new product launch you might want to prepare a version of your site with a new section on the site featuring the new product. The new section would introduce several new pages, updates to existing pages, and new menu items. You want to be able to build out the updated version in a self-contained 'branch' and merge all the changes as a whole when the product is ready to launch. In an election case scenario, you might want to prepare multiple sections; one for each candidate that could win.
  3. Preview your site — When you're building out a new section on your site for launch, you want to preview your entire site, as it will look on the day it goes live. This is effectively content staging on a single site.
  4. Offline browse and publish — Here is a use-case that Pfizer is trying to solve. A sales rep goes to a hospital and needs access to information when there is no wi-fi or a slow connection. The site should be fully functional in offline mode and any changes or forms submitted, should automatically sync and resolve conflicts when the connection is restored.
  5. Content recovery — Even with confirmation dialogs, people delete things they didn’t want to delete. This case is about giving users the ability to “undelete” or recover content that has been deleted from their database.
  6. Audit logs — For compliance reasons, some organizations need all content revisions to be logged, with the ability to review content that has been deleted and connect each action to a specific user so that employees are accountable for their actions in the CMS.

Technical details

All these use cases share a few key traits:

  1. Content needs to be synchronized from one place to another, e.g. from workspace to workspace, from site to site or from frontend to backend
  2. Full revision history needs to be kept
  3. Content revision conflicts needs to be tracked

Much of this started as a single module: Deploy. The Deploy module was first created by Greg Dunlap for Drupal 6 in 2008 for a customer of Palantir. In 2012, Greg handed over maintainership to Dick Olsson. Dick continued to improve Deploy module for Al Jazeera while working at Node One. Later, Dave Hall created a second Drupal 7 version which more significant improvements based on feedback from different users. Today, both Dick and Dave work for Pfizer and have continued to include lessons learned in the Drupal 8 version of the module. After years of experience working on Deploy module and various redesigns, the team has extracted the functionality in a set of modules:

Pfizer content workflow improvements modules

Multiversion

This module does three things: (1) it adds revision support for all content entities in Drupal, not just nodes and block content as provided by core, and (2) it introduces the concept of parent revisions so you can create different branches of your content or site, and (3) it keeps track of conflicts in the revision tree (e.g. when two revisions share the same parent). Many of these features complement the ongoing improvements to Drupal's Entity API.

Replication

Built on top of Multiversion module, this lightweight module reads revision information stored by Multiversion, and uses that to determine what revisions are missing from a given location and lets you replicate content between a source and a target location. The next two modules, Workspace and RELAXed Web Services depend on replication module.

Workspace

This module enables single site content staging and full-site previews. The UI lets you create workspaces and switch between them. With Replication module different workspaces on the same site can behave like different editorial environments.

RELAXed Web Services

This module facilitates cross-site content staging. It provides a more extensive REST API for Drupal with support for UUIDs, translations, file attachments and parent revisions — all important to solve unique challenges with content staging (e.g. UUID and revision information is needed to resolve merge conflicts). The RELAXed Web Services module extends the Replication module and makes it possible to replicate content from local workspaces to workspaces on remote sites using this API.

In short, Multiversion provides the "storage plumbing", whereas Replication, Workspace, and RELAXed Web Services, provide the "transport plumbing".

Deploy

Historically Deploy module has taken care of everything from bottom to top related to content staging. But for Drupal 8 Deploy has been rewritten to just provide a UI on-top of the Workspace and Replication modules. This UI lets you manage content deployments between workspaces on a single site, or between workspaces across sites (if used together with RELAXed Web Services module). The maintainers of the Deploy module have put together a marketing site with more details on what it does: http://www.drupaldeploy.org.

Trash

To handle use case #5 (content recovery) the Trash module was implemented to restore entities marked as deleted. Much like a desktop trash or recycle bin, the module displays all entities from all supported content types where the default revision is flagged as deleted. Restoring creates a newer revision, which is not flagged as deleted.

Synchronizing sites with a battle-tested API

When a Drupal site installs and enables RELAXed Web Services it will look and behave like the REST API from CouchDB. This is a pretty clever trick because it enables us to leverage the CouchDB ecosystem of tools. For example, you can use PouchDB, a JavaScript implementation of CouchDB, to provide a fully-decoupled offline database in the web browser or on a mobile device. Using the same API design as CouchDB also gives you "streaming replication" with instant updates between the backend and frontend. This is how Pfizer implements off-line browse and publish.

Switching between multiple workspaces
This animated gif shows how a content creator can switch between multiple workspaces and get a full-site preview on a single site. In this example the Live workspace is empty, while there has been a lot of content addition on the Stage workspace in.
Deploying a workspace
This animated gif shows how a workspace is deployed from 'Stage' to 'Live' on a single site. In this example the Live workspace is initially empty.

Conclusion

Drupal 8.0 core packed many great improvements, but we didn't focus much on advancing Drupal's content workflow capabilities. As we think about Drupal 8.x and beyond, it might be good to move some of our focus to features like content staging, better audit-ability, off-line support, full-site preview, and more. If you are a content manager, I'd love to hear what you think about better supporting some or all of these use cases. And if you are a developer, I encourage you to take a look at these modules, try them out and let us know what you think.

Thanks to Tim Millwood, Dick Olsson and Nathaniel Catchpole for their feedback on the blog post. Special thanks to Pfizer for contributing to Drupal.

Examples of how to make Drupal outside-in

The authoring experience improvements we made in Drupal 8 appear to be well-received, but that doesn't mean we are done. With semantic versioning in Drupal 8, we can now make more UX changes in follow-up releases of Drupal 8. So now is a good time to start moving Drupal's user experience forward.

The goal of this post is to advance the conversation we started over a month ago in a blog post talking about the concept of turning Drupal outside-in. In today's blog post, we'll show concrete examples of how we could make site building in Drupal easier by embracing the concept of outside-in. We hope to provide inspiration to designers, core contributors and module maintainers for how we can improve the user experience of Drupal 8.2 and beyond.

What is outside-in?

In Drupal you often have to build things from the ground up. If you want to make a list of events you need to wade through 20 different menu options to a user interface with configuration options like "boolean" and "float", build a content type, content, and then a view. Essentially you need to understand everything before you can build anything.

In an "outside-in" experience – or what Kevin OLeary (Director of Design on my team at Acquia) calls Literal UI – you can click anything on the page, edit its configuration in-place, and watch it take effect immediately.

Over the past few years Drupal has adopted a more outside-in approach, the most obvious example being Drupal 8's in-place editing. It enables you to edit what you see with an uninterrupted workflow, faster preview, and removes the need to visit the administration backend before you can start editing.

To evaluate how outside-in can be applied more broadly in order to make Drupal easier to use, Kevin created a few animated gifs.

Example #1: editing menu items

The current inside-out experience

Editing menu options in Drupal has already become more outside-in with contextual links. The contextual links take away some of the guesswork of finding the proper administration UI, but once you arrive at the configuration page there is still a lot of stuff to take in, only some of which is relevant to this task.


The current inside-out experience for editing a menu item in Drupal

The current inside-out experience for editing a menu item: adding a menu link and changing its position involves 2 separate administration pages, 4 page refreshes and more than 400 words of text on the UI.

Anyone familiar with Drupal will recognize the pattern above; you go to an administration UI, make some changes, than you go back to the page to see if it worked. This context switching creates what UX people call "cognitive load". On an administration page you need to remember what was on the site page and vice-versa.

To complete this task you need to:

  1. Find the contextual link for the menu (not simple since it's on the far side of the page)
  2. Choose the correct option "edit menu"
  3. Find and click the "add link" button
  4. Type the menu link name
  5. Type the menu link path beginning with a forward slash
  6. Understand that this change needs to be saved
  7. Scroll to the bottom of the page
  8. Save the link
  9. Find the link in the list of links
  10. Understand that dragging up/down in this abstraction is equivalent to moving right/left in that actual page
  11. Scroll to the bottom of the page
  12. Save the menu

The problem is not just that there are too many pages, clicks, or words, it's that each step away from the actual page introduces new opportunities for confusion, error and repetition. In user testing, we have seen users who were unable to find the contextual link, or to understand which option to choose, or to find the "add link" button, or to add a path, or drag-drop links, or to save before leaving the UI. When these things happen in context, feedback about whether you are "doing it right" is immediate.

The proposed outside-in experience


The proposed outside-in experience for editing a menu item in Drupal

The proposed outside-in experience for editing a menu item. Rather than moving out-of-context to an administration page to get the job done, configuration happens right on the page where you see the effect of each action. You start adding a menu item simply by selecting the menu on the page. Both the menu item and the item's path are in focus to reinforce the connection. As you type a path is proposed from the link text.

Now all you need to do is:

  1. Click the menu on the page
  2. Find and click the "add link" button
  3. Type the name of the menu item
  4. Revise the menu item's path if needed
  5. Drag the link to its new location
  6. Close the drawer

One important aspect of this approach is that all actions that produce a visible change have bi-directional control and bi-directional feedback. In other words, if you can drag something in the configuration drawer you should also be able to drag it on the page, and the changes should happen simultaneously.

Example #2: adding a block to a page

The current inside-out experience

The process of placing a block on a page can be difficult. Once you discover where to go to add a block, which is in itself a challenge, the experience requires a lot of reading and learning, as well as trial and error.


The current inside-out experience for adding a block

The current inside-out experience for adding a block. Not all steps are shown.

To complete this task you need to:

  1. Figure out where to go to place a block
  2. Go to /block-layout
  3. Go to /display-regions to find out where the region is on the page
  4. Go back to /block-layout
  5. Find the region and click "add block"
  6. Find the block you want to place and click "place block"
  7. Configure the block
  8. Read about how blocks can appear on multiple pages and for different content types and roles
  9. Read what content types are
  10. Read what roles are
  11. Read what a "path" is
  12. Read how to find the path for a page
  13. Go back to the page and get its path
  14. Go back to /block-layout and add the path to the visibility settings
  15. Drag your block to the position where you want it
  16. If your blocks are arranged horizontally, learn that "up and down" in the block layout UI will mean "left and right" on the actual page
  17. Find the"back to site" link
  18. Go back to the page to see if you did it right

Eventually you'll use what you just learned, but Drupal makes you learn it first instead of just showing what is immediately necessary. Both the task and the learning can be simplified by bringing the configuration closer to the object you are configuring.

The proposed outside-in experience


The proposed outside-in experience for adding a block

The proposed outside-in experience for adding a block. Everything happens on the actual page rather than on an administration page. Places where things can be added appear on hover. On click they show thumbnails that you can filter with autocomplete.

Now all you need to do is:

  1. Choose where to place the block
  2. Find the block you want to place
  3. Place the block
  4. Close the drawer

The "plus" button, the drop target (blue dotted rectangle) and the autocomplete are all commonly understood patterns used by other software. The task requires no or little explanation as the interactions reveal the process. By starting with selecting the location of where to place the block, we avoid the need for drag-and-drop and the complexity of dragging a block on a page that requires scrolling.

Principles and patterns

These examples show the principle that rather than taking the site builder to a separate administration backend, the experience should begin with the actual page and show how the changes will be seen by the end-user. The patterns shown here are less important. For example, the animated gifs above show two different approaches to the options panel and there are certainly others. The important thing is not yet where the panel comes from or how it looks, but that the following criteria are met:

  • An option panel is triggered by direct interaction.
  • When triggered it only shows options for what you selected.
  • It primarily shows configuration options that produce a visible change to the page. More complex options could be accessed through an 'Advanced options' link.

The ideas in this post are meant to provide some food for thought and become the seed of some patches for Drupal 8.2. Rather than dive right into development, we're looking to the Drupal community to take these designs as a starting point to prototype and user-test more outside-in experiences in Drupal.

The next post in this series will cover outside-in experiences with more complex contexts and the problem of "leaky abstractions". If you don't want to miss the next blog post, make sure to subscribe. In the mean time, I'd love to hear what you think!

Special thanks to Kevin OLeary for advocating outside-in thinking. Thanks to Preston So, Gábor Hojtsy, Angie Byron, Bojhan Somers and Roy Scholten for their feedback.

State of Drupal presentation (February 2016)

I was excited to travel to India a few months ago for DrupalCon, an area where we have a really big opportunity for the growth of Drupal. In keeping with tradition, here are the slides and video from my keynote presentation. You can watch the recording of my keynote (starting at 20:15) or download a copy of my slides (PDF, 158 MB).

The main areas of focus for the talk included Drupal's rapid growth and progress in India, key technology trends driving the future of the web, and how Drupal is responding to these trends. As a call-to-action, I encouraged Drupalists in India to form grassroots communities locally, to become a part of the larger Drupal community conversation, and to port modules from Drupal 7 to Drupal 8 to accelerate its adoption.

Have a look and as always, feel free to leave your thoughts in the comments!

How should you decouple Drupal?

With RESTful web services in Drupal 8 core, Drupal can function as an API-first back end serving browser applications, native applications on mobile devices, in-store displays, even in-flight entertainment systems (Lufthansa is doing so in Drupal 8!), and much more. When building a new website or web application in 2016, you may ask yourself: how should I decouple Drupal? Do I build my website with Drupal's built-in templating layer or do I use a JavaScript framework? Do I need Node.js?

There is a lot of hype around decoupled architectures, so before embarking on a project, it is important to make a balanced analysis. Your choice of architecture has implications on your budget, your team, time to launch, the flexibility for content creators, the ongoing maintenance of your website, and more. In this blog post, I'd like to share a flowchart that can help you decide when to use what technology.

Decoupled decision flowchart

This flowchart shows three things:

First, using coupled Drupal is a perfectly valid option for those who don't need extensive client-side rendering and state management. In this case, you would use Drupal's built-in Twig templating system rather than heavily relying on a JavaScript framework. You would use jQuery to take advantage of limited JavaScript where necessary. Also, with BigPipe in Drupal 8.1, certain use cases that typically needed asynchronous JavaScript can now be done in PHP without slowing down the page (i.e. communication with an external web service delaying the display of user-specific real-time data). The advantage of this approach is that content marketers are not blocked by front-end developers as they assemble their user experiences, thus shortening time to market and reducing investment in ongoing developer support.

Second, if you want all of the benefits of a JavaScript framework without completely bypassing Drupal's HTML generation and all that you get with it, I recommend using progressively decoupled Drupal. With progressive decoupling, you start with Drupal's HTML output, and then use a JavaScript framework to add client-side interactivity on the client side. One of the most visited sites in the world, The Weather Channel (100 million unique visitors per month), does precisely this with Angular 1 layered on top of Drupal 7. In this case, you can enjoy the benefits of having a “decoupled" team made up of both Drupal and JavaScript developers progressing at their own velocities. JavaScript developers can build richly interactive experiences while leaving content marketers free to assemble those experiences without needing a developer's involvement.

Third, whereas fully decoupled Drupal makes a lot of sense when building native applications, for most websites, the leap to fully decoupling is not strictly necessary, though a growing number of people prefer using JavaScript these days. Advantages include some level of independence on the underlying CMS, the ability to tap into a rich toolset around JavaScript (e.g. Babel, Webpack, etc.) and a community of JavaScript front-end professionals. But if you are using a universal JavaScript approach with Drupal, it's also important to consider the drawbacks: you need to ask yourself if you're ready to add more complexity to your technology stack and possibly forgo functionality provided by a more integrated content delivery system, such as layout and display management, user interface localization, and more. Losing that functionality can be costly, increase your dependence on a developer team, and hinder the end-to-end content assembly experience your marketing team expects, among other things.

It's worth noting that over time we are likely to see better integrations between Drupal and the different JavaScript frameworks (e.g. Drupal modules that export their configuration, and SDKs for different JavaScript frameworks that use that configuration on the client-side). When those integrations mature, I expect more people will move towards fully decoupled Drupal.

To be performant, fully decoupled websites using JavaScript employ Node.js on the server to improve initial performance, but in the case of Drupal this is not necessary, as Drupal can do the server-side pre-rendering for you. Many JavaScript developers opt to use Node.js for the convenience of shared rendering across server and client rather than for the specific things that Node.js excels in, like real-time push, concurrent connections, and bidirectional client-server communication. In other words, most Drupal websites don't need Node.js.

Decoupled delivery architectures

In practice, I believe many organizations want to use all of these content delivery options. In certain cases, you want to let your content management system render the experience so you can take full advantage of its features with minimal or no development effort (coupled architecture). But when you need to build a website that needs a much more interactive experience or that integrates with unique devices (i.e. on in-store touch screens), you should be able to use that same content management system's content API (decoupled architecture). Fortunately, Drupal allows you to use either. The beauty of choosing from the spectrum of fully decoupled Drupal, progressively decoupled Drupal, and coupled Drupal is that you can do what makes the most sense in each situation.

Special thanks to Preston So, Alex Bronstein and Wim Leers for their contributions to this blog post. We created at least 10 versions of this flowchart before settling on this one.

A "MAP" for accelerating Drupal 8 adoption

Contributed modules in Drupal deliver the functionality and innovation proprietary content management solutions simply can't match. With every new version of Drupal comes the need to quickly move modules forward from the previous version. For users of Drupal, it's crucial to know they can depend on the availability of modules when considering a new Drupal 8 project or migrating from a previous version.

I'm pleased that many agencies and customers who use Drupal are donating time and attention to maintaining Drupal's module repository and ensuring their contributed modules are upgraded. I believe it's the responsibility of Drupal companies to give back to the community.

I'm proud that Acquia leads by example. It was with great pride that Acquia created a Drupal 8 Module Acceleration Program, or MAP. Led by Acquia's John Kennedy, MAP brings financial, technical and project management assistance to Drupal module maintainers. Acquia kicked off MAP in mid-October and to date we have helped complete production-ready versions of 34 modules. And it is not just any modules; we've been focused on those modules that provide critical pieces of functionality used by most Drupal sites.

When MAP was formed Acquia allocated $500,000 to fund non-Acquia maintainers in the community. In addition, we have so far invested more than 2,500 hours of our own developers' time to support the effort (the equivalent of three full-time developers).

What is impressive to me about MAP is both the focus on mission-critical modules that benefit a huge number of users, as well as the number of community members and agencies involved. John's team is leading a coalition of the best and brightest minds in the Drupal community to address the single biggest obstacle holding Drupal 8 adoption back.

Drupal 8 has already made a significant impact; in the 90 days following the release of Drupal 8.0.0, adoption has outpaced Drupal 7 by more than 200 percent. And as more modules get ported, I expect Drupal 8 adoption to accelerate even more.

Pages

Updates from Dries straight to your mailbox