A roadmap for making Drupal more API-first

In one of my recent blog posts, I articulated a vision for the future of Drupal's web services, and at DrupalCon New Orleans, I announced the API-first initiative for Drupal 8. I believe that there is considerable momentum behind driving the web services initiative. As such, I want to provide a progress report, highlight some of the key people driving the work, and map the proposed vision from the previous blog post onto a rough timeline.

Here is a bird's-eye view of the plan for the next twelve months:

8.2 (Q4 2016) 8.3 (Q2 2017) Beyond 8.3 (2017+)
New REST API capabilities
Waterwheel initial release
New REST API capabilities
JSON API module
GraphQL module?
Entity graph iterator?

New REST API capabilities

Wim Leers (Acquia) and Daniel Wehner (Chapter Three) have produced a comprehensive list of the top priorities for the REST module. We're introducing significant REST API advancements in Drupal 8.2 and 8.3 in order to improve the developer experience and extend the capabilities of the REST API. We've been focused on configuration entity support, simplified REST configuration, translation and file upload support, pagination, and last but not least, support for user login, logout and registration. All this work starts to address differences between core's REST module and various contributed modules like Services and RELAXed Web Services. More details are available in my previous blog post.

Many thanks to Wim Leers (Acquia), Daniel Wehner (Chapter Three), Ted Bowman (Acquia), Alex Pott (Chapter Three), and others for their work on Drupal core's REST modules. Though there is considerable momentum behind efforts in core, we could always benefit from new contributors. Please consider taking a look at the REST module issue queue to help!

Waterwheel initial release

As I mentioned in my previous post, there has been exciting work surrounding Waterwheel, an SDK for JavaScript developers building Drupal-backed applications. If you want to build decoupled applications using a JavaScript framework (e.g. Angular, Ember, React, etc.) that use Drupal as a content repository, stay tuned for Waterwheel's initial release later this year.

Waterwheel aims to facilitate the construction of JavaScript applications that communicate with Drupal. Waterwheel's JavaScript library allows JavaScript developers to work with Drupal without needing deep knowledge of how requests should be authenticated against Drupal, what request headers should be included, and how responses are molded into particular data structures.

The Waterwheel Drupal module adds a new endpoint to Drupal's REST API allowing Waterwheel to discover entity resources and their fields. In other words, Waterwheel intelligently discovers and seamlessly integrates with the content model defined on any particular Drupal 8 site.

A wider ecosystem around Waterwheel is starting to grow as well. Gabe Sullice (Aten Design Group), creator of the Entity Query API module, has contributed an integration of Waterwheel which opens the door to features such as sorts, conditions and ranges. The Waterwheel team welcomes early adopters as well as those working on other REST modules such as JSON API and RELAXed or using native HTTP clients in JavaScript frameworks to add their own integrations to the mix.

Waterwheel is the currently the work of Matt Grill (Acquia) and Preston So (Acquia), who are developing the JavaScript library, and Ted Bowman (Acquia), who is working on the Drupal module.

JSON API module

In conjunction with the ongoing efforts in core REST, parallel work is underway to build a JSON API module which embraces the JSON API specification. JSON API is a particular implementation of REST that provides conventions for resource relationships, collections, filters, pagination, and sorting, in addition to error handling and full test coverage. These conventions help developers build clients faster and encourages reuse of code.

Thanks to Mateu Aguiló Bosch, Ed Faulkner and Gabe Sullice (Aten Design Group), who are spearheading the JSON API module work. The module could be ready for production use by the end of this year and included as an experimental module in core by 8.3. Contributors to JSON API are meeting weekly to discuss progress moving forward.

Beyond 8.3: GraphQL and entity graph iterator

While these other milestones are either certain or in the works, there are other projects gathering steam. Chief among these is GraphQL, which is a query language I highlighted in my Barcelona keynote and allows for clients to tailor the responses they receive based on the structure of the requests they issue.

One of the primary outcomes of the New Orleans web services discussion was the importance of a unified approach to iterating Drupal's entity graph; both GraphQL and JSON API require such an "entity graph iterator". Though much of this is still speculative and needs greater refinement, eventually, such an "entity graph iterator" could enable other functionality such as editable API responses (e.g. aliases for custom field names and timestamp formatters) and a unified versioning strategy for web services. However, more help is needed to keep making progress, and in absence of additional contributors, we do not believe this will land in Drupal until after 8.3.

Thanks to Sebastian Siemssen, who has been leading the effort around this work, which is currently available on GitHub.

Validating our work and getting involved

In order to validate all of the progress we've made, we need developers everywhere to test and experiment with what we're producing. This means stretching the limits of our core REST offerings, trying out JSON API for your own Drupal-backed applications, reporting issues and bugs as you encounter them, and participating in the discussions surrounding this exciting vision. Together, we can build towards a first-class API-first Drupal.

Special thanks to Preston So for contributions to this blog post and to Wim Leers for feedback during its writing.

Drupal is for ambitious digital experiences

What feelings does the name Drupal evoke? Perceptions vary from person to person; where one may describe it in positive terms as "powerful" and "flexible", another may describe it negatively as "complex". People describe Drupal differently not only as a result of their professional backgrounds, but also based on what they've heard and learned.

If you ask different people what Drupal is for, you'll get many different answers. This isn't a surprise because over the years, the answers to this fundamental question have evolved. Drupal started as a tool for hobbyists building community websites, but over time it has evolved to support large and sophisticated use cases.

Perception is everything

Perception is everything; it sets expectations and guides actions and inactions. We need to better communicate Drupal's identity, demonstrate its true value, and manage its perceptions and misconceptions. Words do lead to actions. Spending the time to capture what Drupal is for could energize and empower people to make better decisions when adopting, building and marketing Drupal.

Truth be told, I've been reluctant to define what Drupal is for, as it requires making trade-offs. I have feared that we would make the wrong choice or limit our growth. Over the years, it has become clear that not defining what Drupal is used for leaves more people confused even within our own community.

For example, because Drupal evolved from a simple tool for hobbyists to a more powerful digital experience platform, many people believe that Drupal is now "for the enterprise". While I agree that Drupal is a great fit for the enterprise, I personally never loved that categorization. It's not just large organizations that use Drupal. Individuals, small startups, universities, museums and non-profits can be equally ambitious in what they'd like to accomplish and Drupal can be an incredible solution for them.

Defining what Drupal is for

Rather than using "for the enterprise", I thought "for ambitious digital experiences" was a good phrase to describe what people can build using Drupal. I say "digital experiences" because I don't want to confine this definition to traditional browser-based websites. As I've stated in my Drupalcon New Orleans keynote, Drupal is used to power mobile applications, digital kiosks, conversational user experiences, and more. Today I really wanted to focus on the word "ambitious".

"Ambitious" is a good word because it aligns with the flexibility, scalability, speed and creative freedom that Drupal provides. Drupal projects may be ambitious because of the sheer scale (e.g. The Weather Channel), their security requirements (e.g. The White House), the number of sites (e.g. Johnson & Johnson manages thousands of Drupal sites), or specialized requirements of the project (e.g. the New York MTA powering digital kiosks with Drupal). Organizations are turning to Drupal because it gives them greater flexibility, better usability, deeper integrations, and faster innovation. Not all Drupal projects need these features on day one -- or needs to know about them -- but it is good to have them in case you need them later on.

"Ambitious" also aligns with our community's culture. Our industry is in constant change (responsive design, web services, social media, IoT), and we never look away. Drupal 8 was a very ambitious release; a reboot that took one-third of Drupal's lifespan to complete, but maneuvered Drupal to the right place for the future that is now coming. I have always believed that the Drupal community is ambitious, and believe that attitude remains strong in our community.

Last but not least, our adopters are also ambitious. They are using Drupal to transform their organizations digitally, leaving established business models and old business processes in the dust.

I like the position that Drupal is ambitious. Stating that Drupal is for ambitious digital experiences however is only a start. It only gives a taste of Drupal's objectives, scope, target audience and advantages. I think we'd benefit from being much more clear. I'm curious to know how you feel about the term "for ambitious digital experiences" versus "for the enterprise" versus not specifying anything. Let me know in the comments so we can figure out how to collectively change the perception of Drupal.

PS: I'm borrowing the term "ambitious" from the Ember.js community. They use the term in their tagline and slogan on their main page.

The long path to being understood

I sent an internal note to all of Acquia's 700+ employees today and decided to cross-post it to my blog because it contains a valuable lesson for any startup. One of my personal challenges — both as an Open Source evangelist/leader and entrepreneur — has been to learn to be comfortable with not being understood. Lots of people didn't believe in Open Source in Drupal's early days (and some still don't). Many people didn't believe Acquia could succeed (and some still don't). Something is radically different in software today, and the world is finally understanding and validating that some big shifts are happening. In many cases, an idea takes years to gain general acceptance. Such is the story of Drupal and Acquia. Along the way it can be difficult to deal with the naysayers and rejections. If you ever have an idea that is not understood, I want you to think of my story.

Team,

This week, Acquia got a nice mention on Techcrunch in an article written by Jake Flomenberg, a partner at Accel Partners. For those of you who don't know Accel Partners, they are one of the most prominent venture capital investors and were early investors in companies like Facebook, Dropbox, Slack, Etsy, Atlassian, Lynda.com, Kayak and more.

The article, called "The next wave in software is open adoption software", talks about how the enterprise IT stack is being redrawn atop powerful Open Source projects like MongoDB, Hadoop, Drupal and more. Included in the article is a graph that shows Acquia's place in the latest wave of change to transform the technology landscape, a place showing our opportunity is bigger than anything before as the software industry migrated from mainframes to client-server, then SaaS/PaaS and now - to what Flomenberg dubs, the age of Open Adoption Software.

Waves of software adoption

It's a great article, but it isn't new to any of us per se – we have been promoting this vision since our start nine years ago and we have seen over and over again how Open Source is becoming the dominant model for how enterprises build and deliver IT. We have also shown that we are building a successful technology company using Open Source.

Why then do I feel compelled to share this article, you ask? The article marks a small but important milestone for Acquia.

We started Acquia to build a new kind of company with a new kind of business model, a new innovation model, all optimized for a new world. A world where businesses are moving most applications into the cloud, where a lot of software is becoming Open Source, where IT infrastructure is becoming a metered utility, and where data-driven services make or break business results.

We've been steadily executing on this vision; it is why we invest in Open Source (e.g. Drupal), cloud infrastructure (e.g. Acquia Cloud and Site Factory), and data-centric business tools (e.g. Acquia Lift).

In my 15+ years as an Open Source evangelist, I've argued with thousands of people who didn't believe in Open Source. In my 8+ years as an entrepreneur, I've talked to thousands of business people and dozens of investors who didn't understand or believe in Acquia's vision. Throughout the years, Tom and I have presented Acquia's vision to many investors – some have bought in and some, like Accel, have not (for various reasons). I see more and more major corporations and venture capital firms coming around to Open Source business models every day. This trend is promising for new Open Source companies; I'm proud that Acquia has been a part of clearing their path to being understood.

When former skeptics become believers, you know you are finally being understood. The Techcrunch article is a small but important milestone because it signifies that Acquia is finally starting to be understood more widely. As flattering as the Techcrunch article is, true validation doesn't come in the form of an article written by a prominent venture capitalist; it comes day-in and day-out by our continued focus and passion to grow Drupal and Acquia bit by bit, one successful customer at a time.

Building a new kind of company like we are doing with Acquia is the harder, less-traveled path, but we always believed it would be the best path for our customers, our communities, and ultimately, our world. Success starts with building a great team that not only understands what we do, but truly believes in what we do and remains undeterred in its execution. Together, we can build this new kind of company.

--
Dries Buytaert
Founder and Project Lead, Drupal
Co-founder and Chief Technology Officer, Acquia

Microsoft buys LinkedIn: the value of data

In my latest SXSW talk, I showed a graphic of each of the major technology giants to demonstrate how much of our user data each company owned.

Microsoft linkedin data

I said they won't stop until they know everything about us. Microsoft just bought LinkedIn, so here is what happened:

By acquiring the world's largest professional social network, Microsoft gets immediate access to data from more than 433 million LinkedIn members. Microsoft fills out the "social graph" and "interests" circles. There is speculation over what Microsoft will do with LinkedIn over time, but here is what I think is most likely:

  • With LinkedIn, Microsoft could build out its Microsoft Dynamics CRM business to reinvent the sales and marketing process, helping the company compete more directly with SalesForce.
  • LinkedIn could allow Microsoft to implement a "Log in with LinkedIn" system similar to Facebook Connect. Microsoft could turn LinkedIn profiles into a cross-platform business identity to better compete with Google and Facebook.
  • LinkedIn could allow Microsoft to build out Cortana, a workplace-tailored digital assistant. One scenario Microsoft referenced was walking into a meeting and getting a snapshot of each attendee based on his or her LinkedIn profile. This capability will allow Microsoft to better compete against virtual assistants like Google Now, Apple Siri and Amazon Echo.
  • LinkedIn could be integrated in applications like Outlook, Skype, Office, and even Windows itself. Buying LinkedIn helps Microsoft limit how Facebook and Google are starting to get into business applications.

Data is eating the world

In the past I wrote that data, not software, is eating the world. The real value in technology comes less and less from software and more and more from data. As most businesses are moving applications into the cloud, a lot of software is becoming free, IT infrastructure is becoming a metered utility, and data is what is really makes or breaks business results. Here is one excerpt from my post: "As value shifts from software to the ability to leverage data, companies will have to rethink their businesses. In the next decade, data-driven, personalized experiences will continue to accelerate, and development efforts will shift towards using contextual data.". This statement is certainly true in Microsoft / LinkedIn's case.

Microsoft linkedin graphs
Source: Microsoft.

If this deal shows us anything, it's about the value of user data. Microsoft paid more than $60 per registered LinkedIn user. The $26.2 billion price tag values LinkedIn at about 91 times earnings, and about 7 percent of Microsoft's market cap. This is a very bold acquisition. You could argue that this is too hefty a price tag for LinkedIn, but this deal is symbolic of Microsoft rethinking its business strategy to be more data and context-centric. Microsoft sees that the future for them is about data and I don't disagree with that. While I believe acquiring LinkedIn is a right strategic move for Microsoft, I'm torn over whether or not Microsoft overpaid for LinkedIn. Maybe we'll look back on this acquisition five years from now and find that it wasn't so crazy, after all.

Investing in Open Source startups

From the day we started Acquia, we had big dreams: we wanted to build a successful company, while giving back to the Open Source community. Michael Skok was our first investor in Acquia and instrumental in making Acquia one of the largest Open Source companies in the world, creating hundreds of careers for people passionate about Open Source. This week, Michael and his team officially announced a new venture firm called _Underscore.VC. I'm excited to share that I joined _Underscore.VC as a syndicate lead for the "Open Source _Core".

I'm very passionate about Open Source and startups, and want to see more Open Source startups succeed. In my role as the syndicate lead for the Open Source _Core, I can help other Open Source entrepreneurs raise money, get started and scale their companies and Open Source projects.

Does that mean I'll be leaving Drupal or Acquia? No. I'll continue as the lead of the Drupal project and the CTO of Acquia. Drupal and Acquia continue to be my full-time focus. I have been advising entrepreneurs and startups for the last 5+ years, and have been a moderately active angel investor the past two years. Not much, if anything, will change about my day-to-day. _Underscore.VC gives me a better platform to advise and invest, give back and help others succeed with Open Source startups. It's a chance to amplify the "do well and do good" mantra that drives me.

Mautic and the power of syndicates

While Michael, the _Underscore.VC team and I have been working on _Underscore.VC for quite some time, I'm excited to share that on top of formally launching this week, they've unveiled a $75 million fund, as well as our first seed investment. This first investment is in Mautic, an Open Source marketing automation company.

Mautic is run by David Hurley, who I've known since he was a community manager at Joomla!. I've had the opportunity to watch David grow for many months. His resourcefulness, founding and building the Mautic product and Open Source community impressed me.

The Mautic investment is a great example of _Underscore.VC's model in action. Unlike a traditional firm, _Underscore.VC co-invests with a group of experts, called a syndicate, or in the case of _Underscore.VC a "_Core". Each _Core has one or more leads that bring companies into the process and gather the rest of the investors to form a syndicate.

As the lead of the Open Source _Core, I helped pull together a group of investors with expertise in Open Source business models, marketing automation, and SaaS. The list of people includes Larry Augustin (CEO of SugarCRM), Gail Goodman (CEO of Constant Contact), Erica Brescia (Co-Founder and COO of Bitnami), Andrew Aitken (Open Source Lead at Wipro) and more. Together with _Underscore.VC, we made a $600,000 seed investment in Mautic. In addition to the funding, Mautic will get access to a set of world-class advisors invested in helping them succeed.

I personally believe the _Underscore.VC model has the power to transform venture capital. Having raised over $180 million for Acquia, I can tell you that fundraising is no walk in the park. Most investors still don't understand Open Source business models. To contrast, our Open Source _Core group understands Open Source deeply; we can invest time in helping Mautic acquire new customers, recruit great talent familiar with Open Source, partner with the right companies and navigate the complexities of running an Open Source business. With our group's combined expertise, I believe we can help jumpstart Mautic and reduce their learnings by one to two years.

It's also great for us as investors. By combining our operating experience, we hope to attract entrepreneurs and startups that most investors may not get the opportunity to back. Furthermore, the _Core puts in money at the same valuation and terms as _Underscore.VC, so we can take advantage of the due diligence horsepower that _Underscore.VC provides. The fact that _Underscore.VC can write much larger checks is also mutually beneficial to the _Core investor and the entrepreneur; it increases the chances of the entrepreneur succeeding.

If you're starting an Open Source business, or if you're an angel investor willing to co-invest in the Open Source _Core, feel free to reach out to me or to get in touch with _Underscore.VC.

Updates from Dries straight to your mailbox