In an earlier blog post, I looked at the web services solutions available in Drupal 8 and compared their strengths and weaknesses. That blog post was intended to help developers choose between different solutions when building Drupal 8 sites. In this blog post, I want to talk about how to advance Drupal's web services beyond Drupal 8.1 for the benefit of Drupal core contributors, module creators and technical decision-makers.
Moreover, newer headless content-as-a-service solutions (e.g. Contentful, Prismic.io, Backand and CloudCMS) have entered the market and represent a widening interest in content repositories enabling more flexible content delivery. They provide content modeling tools, easy-to-use tools to construct REST APIs, and SDKs for different programming languages and client-side frameworks.
In my view, we need to do the following, which I summarize in each of the following sections: (1) facilitate a single robust REST module in core; (2) add functionality to help web services modules more easily query and manipulate Drupal's entity graph; (3) incorporate GraphQL and JSON API out of the box; and (4) add SDKs enabling easy integration with Drupal. Though I shared some of this in my DrupalCon New Orleans keynote, I wanted to provide more details in this blog post. I'm hoping to discuss this and revise it based on feedback from you.
One great REST module in core
While core REST can be enabled with only a few configuration changes, the full extent of possibilities in Drupal is only unlocked either when leveraging modules which add to or work alongside core REST's functionality, such as Services or RELAXed, or when augmenting core REST's capabilities with additional resources to interact with (by providing corresponding plugins) or using other custom code.
Having such disparate REST modules complicates the experience. These REST modules have overlapping or conflicting feature sets, which are shown in the following table.
|Feature||Core REST||RELAXed||Services||Ideal core REST|
|Content entity CRUD||Yes||Yes||Yes||Yes|
|Configuration entity CRUD||Create resource plugin (issue)||Create resource plugin||Yes||Yes|
|Custom resources||Create resource plugin||Create resource plugin||Create Services plugin||Possible without code|
|Custom routes||Create resource plugin or Views REST export (GET)||Create resource plugin||Configurable route prefixes||Possible without code|
|Translations||Not yet (issue)||Yes||Create Services plugin||Yes|
|Revisions||Create resource plugin||Yes||Create Services plugin||Yes|
|File attachments||Create resource plugin||Yes||Create Services plugin||Yes|
|Authenticated user resources (log in/out, password reset)||Not yet (issue)||No||User login and logout||Yes|
I would like to see a convergence where all of these can be achieved in Drupal core with minimal configuration and minimal code.
Working with Drupal's entity graph
Recently, a discussion at DrupalCon New Orleans with key contributors to the core REST modules, maintainers of important contributed web services modules, and external observers led to a proposed path forward for all of Drupal's web services.
Buried inside Drupal is an "entity graph" over which different API approaches like traditional REST, JSON API, and GraphQL can be layered. These varied approaches all traverse and manipulate Drupal's entity graph, with differences solely in the syntax and features made possible by that syntax. Unlike core's REST API which only returns a single level (single entity or lists of entities), GraphQL and JSON API can return multiple levels of nested entities as the result of a single query. To better understand what this means, have a look at the GraphQL demo video I shared in my DrupalCon Barcelona keynote.
What we concluded at DrupalCon New Orleans is that Drupal's GraphQL and JSON API implementations require a substantial amount of custom code to traverse and manipulate Drupal's entity graph, that there was a lot of duplication in that code, and that there is an opportunity to provide more flexibility and simplicity. Therefore, it was agreed that we should first focus on building an "entity graph iterator" that can be reused by JSON API, GraphQL, and other modules.
This entity graph iterator would also enable manipulation of the graph, e.g. for aliasing fields in the graph or simplifying the structure. For example, the difference between Drupal's "base fields" and "configured fields" is irrelevant to an application developer using Drupal's web services API, but Drupal's responses leak this internal distinction by prefixing configured fields with
field_ (see the left column in the table below). By the same token, all fields, even if they carry single values, expose the verbosity of Drupal's typed data system by being presented as arrays (see the left column in the table below). While there are both advantages and disadvantages to exposing single-value fields as arrays, many developers prefer more control over the output or the ability to opt into simpler outputs.
A good Drupal entity graph iterator would simplify the development of Drupal web service APIs, provide more flexibility over naming and structure, and eliminate duplicate code.
|Current core REST (shortened response)||Ideal core REST (shortened response)|
GraphQL and JSON API in core
While both JSON API and GraphQL are preferred over traditional REST due to their ability to provide nested entity relationships, GraphQL goes a step further than JSON API by facilitating explicitly client-driven queries, in which the client dictates its data requirements.
SDKs to consume web services
While a unified REST API and support for GraphQL and JSON API would dramatically improve Drupal as a web services back end, we need to be attentive to the needs of consumers of those web services as well by providing SDKs and helper libraries for developers new to Drupal.
An SDK could make it easy to retrieve an article node, modify a field, and send it back without having to learn the details of Drupal's particular REST API implementation or the structure of Drupal's underlying data storage. For example, this would allow front-end developers to not have to deal with the details of single- versus multi-value fields, optional vs required fields, validation errors, and so on. As an additional example, incorporating user account creation and password change requests into decoupled applications would empower front-end developers building these forms on a decoupled front end such that they would not need to know anything about how Drupal performs user authentication.
I believe that it is important to have first-class web services in Drupal out of the box in order to enable top-notch APIs and continue our evolution to become API-first.
In parallel with our ongoing work on shoring up our REST module in core, we should provide the underpinnings for even richer web services solutions in the future. With reusable helper functionality that operates on Drupal's entity graph available in core, we open the door to GraphQL, JSON API, and even our current core REST implementation eventually relying on the same robust foundation. Both GraphQL and JSON API could also be promising modules in core. Last but not least, SDKs like Hydrant that empower developers to work with Drupal without learning its complexities will further advance our web services.
Collectively, these tracks of work will make Drupal uniquely compelling for application developers within our own community and well beyond.
Special thanks to Preston So for contributions to this blog post and to Moshe Weitzman, Kyle Browning, Kris Vanderwater, Wim Leers, Sebastian Siemssen, Tim Millwood, Ted Bowman, and Mateu Aguiló Bosch for their feedback during its writing.
The battle for the marketing cloud just got way more interesting. This week, Salesforce announced its acquisition of Demandware for $2.8B in cash. It will enable Salesforce to offer a "Commerce Cloud" alongside its sales and marketing solutions.
The large platform companies like Oracle and Adobe are trying to own the digital customer experience market from top to bottom by acquiring and integrating together tools for marketing, commerce, customer support, analytics, mobile apps, and more. Oracle's acquisition of Eloqua, SAP's acquisition of hybris and Salesforce's acquisitions of ExactTarget were earlier indicators of market players consolidating SaaS apps for customer experience onto their platforms.
In my view, the Demandware acquisition is an interesting strategic move for Salesforce that aligns them more closely as a competitor to marketing stack mega-vendors such as Adobe, Oracle and IBM. Adding a commerce solution to its suite, makes it easier for Salesforce's customers to build an integrated experience and see what their customers are buying. There are advantages to integrated solutions that have a single system of record about the customer. The Demandware acquisition also makes sense from a technology point of view; there just aren't many Java-based commerce platforms that are purely SaaS-based, that can operate at scale, and that are for sale.
However, we've also seen this movie before. When big companies acquire smaller, innovative companies, over time the innovation goes away in favor of integration. Big companies can't innovate fast enough, and the suite lock-in only benefits the vendor.
There is a really strong case to be made for a best-of-breed approach where you choose and integrate the best software from different vendors. This is a market that literally changes too much and too fast for any organization to buy into a single mega-platform. From my experience talking to hundreds of customer organizations, most prefer an open platform that integrates different solutions and acts as an orchestration hub. An open platform ultimately presents more freedom for customers to build the exact experiences they want. Open Source solutions, like Drupal, that have thousands of integrations, allow organizations to build these experiences in less time, with a lower overall total cost of ownership, more flexibility and faster innovation.
Adobe clearly missed out on buying Demandware, after it missed out on buying Hybris years ago. Demandware would have fit in Adobe's strategy and technology stack. Now Adobe might be the only mega-platform that doesn't have an embedded commerce capability. More interestingly, there don't appear to be large independent commerce operators left to buy.
I continue to believe there is a great opportunity for new independent commerce platforms, especially now Salesforce and Demandware will spend the next year or two figuring out the inevitable challenges of integrating their complex software solutions. I'd love to see more commerce platforms emerge, especially those with a modern micro-services based architecture, and an Open Source license and innovation model.
The Drupal community is very special because of its culture of adapting to change, determination and passion, but also its fun and friendship. It is a combination that is hard to come by, even in the Open Source world. Our culture enabled us to work through really long, but ground-breaking release cycles, which also prompted us to celebrate the release of Drupal 8 with 240 parties around the world.
Throughout Drupal's 15 years history, that culture has served us really well. As the larger industry around us continues to change -- see my DrupalCon New Orleans keynote for recent examples -- we have been able to evolve Drupal accordingly. Drupal has not only survived massive changes in our industry; it has also helped drive them. Very few open source projects are 15 years old and continue to gain momentum.
Drupal 8 is creating new kinds of opportunities for Drupal. For example, who could have imagined that Lufthansa would be using Drupal 8 to build its next-generation in-flight entertainment system? Drupal 8 changes the kind of end-user experiences people can build, how we think about Drupal, and what kind of people we'll attract to our community. I firmly believe that these changes are positive for Drupal, increase Drupal's impact on the world, and grow the opportunity for our commercial ecosystem.
To seize the big opportunity ahead of us and to adjust to the changing environment, it was the Drupal Association's turn to adapt and carefully realign the Drupal Association's strategic focus.
The last couple of years the Drupal Association invested heavily in Drupal.org to support the development and the release of Drupal 8. Now Drupal 8 is released, the Drupal Association's Board of Directors made the strategic decision to shift some focus from the "contribution journey" to the "evaluator's adoption journey" -- without compromising our ability to build and maintain the Drupal software. The Drupal Association will reduce its efforts on Drupal.org's collaboration tools and expand its efforts to grow Drupal's adoption and to build a larger ecosystem of technology partners.
We believe this is not only the right strategic focus at this point in Drupal 8's lifecycle, but also a necessary decision. While the Drupal Association's revenues continued to grow at a healthy pace, we invested heavily, and exhausted our available reserves supporting the Drupal 8 release. As a result, we have to right-size the organization, balance our income with our expenses, and focus on rebuilding our reserves.
In a blog post today, we provide more details on why we made these decisions and how we will continue to build a healthy long-term organization. The changes we made today help ensure that Drupal will gain momentum for decades to come. We could not make this community what it is without the participation of each and every one of you. Thanks for your support!
This is a time of transition for the Drupal Association. As you might have read on the Drupal Association blog, Holly Ross, our Executive Director, is moving on. Megan Sanicki, who has been with the Drupal Association for almost 6 years, and was working alongside Holly as the Drupal Association's COO, will take over Holly's role as the Executive Director.
Open source stewardship is not easy but in the 3 years Holly was leading the Drupal Association, she lead with passion, determination and transparency. She operationalized the Drupal Association and built a team that truly embraces its mission to serve the community, growing that team by over 50% over three years of her tenure. She established a relationship with the community that wasn't there before, allowing the Drupal Association to help in new ways like supporting the Drupal 8 launch, providing test infrastructure, implementing the Drupal contribution credit system, and more. Holly also matured our DrupalCon, expanding its reach to more users with conferences in Latin America and India. She also executed the Drupal 8 Accelerate Fund, which allowed direct funding of key contributors to help lead Drupal 8 to a successful release.
Holly did a lot for Drupal. She touched all of us in the Drupal community. She helped us become better and work closer together. It is sad to see her leave, but I'm confident she'll find success in future endeavors. Thanks, Holly!
Megan, the Drupal Association staff and the Board of Directors are committed to supporting the Drupal project. In this time of transition, we are focused on the work that Drupal Association must do and looking at how to do that in a sustainable way so we can support the project for many years to come.
Last year around this time, I wrote that The Big Reverse of Web would force a major re-architecture of the web to bring the right information, to the right person, at the right time, in the right context. I believe that conversational interfaces like Amazon Echo are further proof that the big reverse is happening.
New user experience and distribution platforms only come along every 5-10 years, and when they do, they cause massive shifts in the web's underlying technology. The last big one was mobile, and the web industry adapted. Conversational interfaces could be the next user experience and distribution platform – just look at Amazon Echo (aka Alexa), Facebook's messenger or Microsoft's Conversation-as-a-Platform.
Today, hardly anyone questions whether to build a mobile-optimized website. A decade from now, we might be saying the same thing about optimizing digital experiences for voice or chat commands. The convenience of a customer experience will be a critical key differentiator. As a result, no one will think twice about optimizing their websites for multiple interaction patterns, including conversational interfaces like voice and chat. Anyone will be able to deliver a continuous user experience across multiple channels, devices and interaction patterns. In some of these cross-channel experiences, users will never even look at a website. Conversational interfaces let users disintermediate the website by asking anything and getting instant, often personalized, results.
To prototype this future, my team at Acquia built a fully functional demo based on Drupal 8 and recorded a video of it. In the demo video below, we show a sample supermarket chain called Gourmet Market. Gourmet Market wants their customers to not only shop online using their website, but also use Echo or push notifications to do business with them.
We built an Alexa integration module to connect Alexa to the Gourmet Market site and to answer questions about sale items. For example, you can speak the command: "Alexa, ask Gourmet Market what fruits are on sale today". From there, Alexa would make a call to the Gourmet Market website, finding what is on sale in the specified category and pull only the needed information related to your ask.
On the website's side, a store manager can tag certain items as "on sale", and Alexa's voice responses will automatically and instantly reflect those changes. The marketing manager needs no expertise in programming -- Alexa composes its response by talking to Drupal 8 using web service APIs.
The demo video also shows how a site could deliver smart notifications. If you ask for an item that is not on sale, the Gourmet Market site can automatically notify you via text once the store manager tags it as "On Sale".
From a technical point of view, we've had to teach Drupal how to respond to a voice command, otherwise known as a "Skill", coming into Alexa. Alexa Skills are fairly straightforward to create. First, you specify a list of "Intents", which are basically the commands you want users to run in a way very similar to Drupal's routes. From there, you specify a list of "Utterances", or sentences you want Echo to react to that map to the Intents. In the example of Gourmet Market above, the Intents would have a command called
GetSaleItems. Once the command is executed, your Drupal site will receive a webhook callback on
/alexa/callback with a payload of the command and any arguments. The Alexa module for Drupal 8 will validate that the request really came from Alexa, and fire a Drupal Event that allows any Drupal module to respond.
It's exciting to think about how new user experiences and distribution platforms will change the way we build the web in the future. As I referenced in Drupalcon New Orleans keynote, the Drupal community needs to put some thought into how to design and build multichannel customer experiences. Voice assistance, chatbots or notifications are just one part of the greater equation. If you have any further thoughts on this topic, please share them in the comments.