A roadmap for making Drupal more API-first

In one of my recent blog posts, I articulated a vision for the future of Drupal's web services, and at DrupalCon New Orleans, I announced the API-first initiative for Drupal 8. I believe that there is considerable momentum behind driving the web services initiative. As such, I want to provide a progress report, highlight some of the key people driving the work, and map the proposed vision from the previous blog post onto a rough timeline.

Here is a bird's-eye view of the plan for the next twelve months:

8.2 (Q4 2016) 8.3 (Q2 2017) Beyond 8.3 (2017+)
New REST API capabilities
Waterwheel initial release
New REST API capabilities
JSON API module
GraphQL module?
Entity graph iterator?

New REST API capabilities

Wim Leers (Acquia) and Daniel Wehner (Chapter Three) have produced a comprehensive list of the top priorities for the REST module. We're introducing significant REST API advancements in Drupal 8.2 and 8.3 in order to improve the developer experience and extend the capabilities of the REST API. We've been focused on configuration entity support, simplified REST configuration, translation and file upload support, pagination, and last but not least, support for user login, logout and registration. All this work starts to address differences between core's REST module and various contributed modules like Services and RELAXed Web Services. More details are available in my previous blog post.

Many thanks to Wim Leers (Acquia), Daniel Wehner (Chapter Three), Ted Bowman (Acquia), Alex Pott (Chapter Three), and others for their work on Drupal core's REST modules. Though there is considerable momentum behind efforts in core, we could always benefit from new contributors. Please consider taking a look at the REST module issue queue to help!

Waterwheel initial release

As I mentioned in my previous post, there has been exciting work surrounding Waterwheel, an SDK for JavaScript developers building Drupal-backed applications. If you want to build decoupled applications using a JavaScript framework (e.g. Angular, Ember, React, etc.) that use Drupal as a content repository, stay tuned for Waterwheel's initial release later this year.

Waterwheel aims to facilitate the construction of JavaScript applications that communicate with Drupal. Waterwheel's JavaScript library allows JavaScript developers to work with Drupal without needing deep knowledge of how requests should be authenticated against Drupal, what request headers should be included, and how responses are molded into particular data structures.

The Waterwheel Drupal module adds a new endpoint to Drupal's REST API allowing Waterwheel to discover entity resources and their fields. In other words, Waterwheel intelligently discovers and seamlessly integrates with the content model defined on any particular Drupal 8 site.

A wider ecosystem around Waterwheel is starting to grow as well. Gabe Sullice (Aten Design Group), creator of the Entity Query API module, has contributed an integration of Waterwheel which opens the door to features such as sorts, conditions and ranges. The Waterwheel team welcomes early adopters as well as those working on other REST modules such as JSON API and RELAXed or using native HTTP clients in JavaScript frameworks to add their own integrations to the mix.

Waterwheel is the currently the work of Matt Grill (Acquia) and Preston So (Acquia), who are developing the JavaScript library, and Ted Bowman (Acquia), who is working on the Drupal module.

JSON API module

In conjunction with the ongoing efforts in core REST, parallel work is underway to build a JSON API module which embraces the JSON API specification. JSON API is a particular implementation of REST that provides conventions for resource relationships, collections, filters, pagination, and sorting, in addition to error handling and full test coverage. These conventions help developers build clients faster and encourages reuse of code.

Thanks to Mateu Aguiló Bosch, Ed Faulkner and Gabe Sullice (Aten Design Group), who are spearheading the JSON API module work. The module could be ready for production use by the end of this year and included as an experimental module in core by 8.3. Contributors to JSON API are meeting weekly to discuss progress moving forward.

Beyond 8.3: GraphQL and entity graph iterator

While these other milestones are either certain or in the works, there are other projects gathering steam. Chief among these is GraphQL, which is a query language I highlighted in my Barcelona keynote and allows for clients to tailor the responses they receive based on the structure of the requests they issue.

One of the primary outcomes of the New Orleans web services discussion was the importance of a unified approach to iterating Drupal's entity graph; both GraphQL and JSON API require such an "entity graph iterator". Though much of this is still speculative and needs greater refinement, eventually, such an "entity graph iterator" could enable other functionality such as editable API responses (e.g. aliases for custom field names and timestamp formatters) and a unified versioning strategy for web services. However, more help is needed to keep making progress, and in absence of additional contributors, we do not believe this will land in Drupal until after 8.3.

Thanks to Sebastian Siemssen, who has been leading the effort around this work, which is currently available on GitHub.

Validating our work and getting involved

In order to validate all of the progress we've made, we need developers everywhere to test and experiment with what we're producing. This means stretching the limits of our core REST offerings, trying out JSON API for your own Drupal-backed applications, reporting issues and bugs as you encounter them, and participating in the discussions surrounding this exciting vision. Together, we can build towards a first-class API-first Drupal.

Special thanks to Preston So for contributions to this blog post and to Wim Leers for feedback during its writing.

Comments

David (not verified):

When I see statements like, "Contributors to JSON API are meeting weekly to discuss progress moving forward," it would be good to include information on how others can get involved. I didn't see that included.

July 07, 2016
dawehner (not verified):
July 08, 2016
User (not verified):

Until you do all this, I might consider to test drive Drupal 8. I thought it was already Rest-ready. For me, Drupal reached its glory at 7. For me now, for new projects (specially those of data management nature) I use Rails or NodeJS, bypassing all Drupal's internal logic. Time will prove how successful Drupal 8 will be, you have to consider making it faster for authenticated use.

July 07, 2016
User (not verified):

Hello Dries, I feel that I am not being totally fair with my criticism above. I know you guys put a lot of thinking and effort into making Drupal 8. I cannot deny that what holds me back is all the learning of the new stuff in 8, lack of fully functional contrib and mostly performance for authenticated users. Drupal in general is great on the back-end. I hope in the future, thoughts will be invested into making it solid rock on the back-end (Restful resources etc..) and into making it easily PnP on the decoupled (JS/Angualr/jQuery driven) front-end. This is my vision for a successful Drupal 8 (with clean-up of perhaps avoidable SQL queries on the server).

July 08, 2016
Wim Leers (not verified):

Drupal 8 is actually significantly faster than Drupal 7 in many ways. Remember that when you test Drupal 7 core's authenticated performance, you also need to add Views at minimum, and probably a bunch of contrib modules.

Drupal 8 has caching enabled by default for all entity types as well as Views (where it makes an enormous difference — this was one of the top reasons for "Drupal 7 is slow" complaints), it comes with Dynamic Page Cache (https://www.drupal.org/documentation/modules/dynamic_page_cache) built-in and enabled by default (which is like authcache in Drupal 7, but does not require deep Drupal knowledge to set up) and has the ability to serve pages using BigPipe (https://www.drupal.org/documentation/modules/big_pipe).

Yes, it's easy to come up with a benchmark that shows Drupal 8 is slow for authenticated users. But almost without exception, it's comparing apples to oranges.

Most Drupal 8 sites will actually load *faster* than Drupal 7 sites: they're structured to ensure the browser can start rendering much sooner, even when server response times look a bit high.

July 12, 2016
dawehner (not verified):

Should experimental modules get into core and then continued to be developed in contrib and synced over from time to time? This could give us the best of both worlds . I think at least Mateu would prefer that way.

Can we please stop talking about the type graph as a thing as long it's not a thing yet? Converting between the various domains of types data, type graph, and actual value graph is not at all fleshed out yet. Mentioning in a post which talks about 8.x improvements IMHO gives the wrong impression.

Hint: I also work mostly for someone and they also support me in contributing.

July 07, 2016
Dries:

Daniel, let me reply to your "credit hint" first as that is important to me. I'm more than happy to update the blog post to credit your sponsor -- just let me know who to credit. For what it is worth, I checked several of your comments on the REST issues and I didn't see any organizational credit. Sorry if I missed it. I might not have looked careful enough. I'm a big champion of highlighting organizations that invest in Drupal core; from introducing and pushing the credit system, to showing company logos in my keynotes and presentations around the world. I'd like to do more of that on my blog as well. Let me know who to credit and I'll update the blog post immediately. Thank you!

July 08, 2016
dawehner (not verified):

Thank you for taking that into account!

No idea which REST issues these has been, but its the same as Alex, Chapter Three.

July 08, 2016
Dries:

Updated the blog post accordingly. Thanks Chapter Three! :-)

July 08, 2016
Shena (not verified):

As a Drupal developer, I am really looking forward to Waterwheel and Jason API. Since I am working on Angular.JS and Node.JS too of late, these latest developments should help me bring best of both the worlds together - Drupal as backend and JS tech stack for front-end. Exciting times ahead.

July 08, 2016
Sebastian Siemssen (not verified):

Daniel and me have started pondering and working on a prototype for the type graph at DevDays. There is still a lot to be fleshed out indeed. The concept is not ready yet to be on a road map but it's worth raising some awareness for it. I am looking forward to the next meeting and really really hope that I can find some time in the meantime to make the prototype worth checking out.

July 08, 2016
Dries:

I tried to do exactly that in my blog post; raise some awareness for the type graph idea, while being very explicit about it being speculative and in need of more refinement. Thanks for re-iterating that in your comment, Sebastian. I hope you find the time or sponsorship because I'm *very* excited about GraphQL. Writing about it publicly could help with finding a sponsor! Hint: Anyone that wants to sponsor Sebastian to work on improving Drupal 8's GraphQL support?

July 08, 2016
Sebastian Siemssen (not verified):

Thanks so much Dries! I am also extremely excited about GraphQL and want to push forward. We are currently working on a Client project fully based on GraphQL, Relay and React in the front-end with Drupal 8 as a back-end. Also see this session proposal for further reference: https://events.drupal.org/dublin2016/sessions/no-compromises-–-react-relay-and-graphql-drupal-8. We are currently working on other parts of the Project but will soon start integrating the Druapl 8 GraphQL Backend which will involve a lot of on-budget work on the GraphQL module. I can't wait for that to start.

July 08, 2016
Sebastian Siemssen (not verified):

Mhmh. I think we just unveiled a bug in Drupal's text formats when automaically web addresses into links. It seems to cut off at multiple hyphens :P.

July 08, 2016

Updates from Dries straight to your mailbox