Digital Distributors vs Open Web: who will win?

I've spent a fair amount of time thinking about how to win back the Open Web, but in the case of digital distributors (e.g. closed aggregators like Facebook, Google, Apple, Amazon, Flipboard) superior, push-based user experiences have won the hearts and minds of end users, and enabled them to attract and retain audience in ways that individual publishers on the Open Web currently can't.

In today's world, there is a clear role for both digital distributors and Open Web publishers. Each needs the other to thrive. The Open Web provides distributors content to aggregate, curate and deliver to its users, and distributors provide the Open Web reach in return. The user benefits from this symbiosis, because it's easier to discover relevant content.

As I see it, there are two important observations. First, digital distributors have out-innovated the Open Web in terms of conveniently delivering relevant content; the usability gap between these closed distributors and the Open Web is wide, and won't be overcome without a new disruptive technology. Second, the digital distributors haven't provided the pure profit motives for individual publishers to divest their websites and fully embrace distributors.

However, it begs some interesting questions for the future of the web. What does the rise of digital distributors mean for the Open Web? If distributors become successful in enabling publishers to monetize their content, is there a point at which distributors create enough value for publishers to stop having their own websites? If distributors are capturing market share because of a superior user experience, is there a future technology that could disrupt them? And the ultimate question: who will win, digital distributors or the Open Web?

I see three distinct scenarios that could play out over the next few years, which I'll explore in this post.

Digital Distributors vs Open Web: who will win?

This image summarizes different scenarios for the future of the web. Each scenario has a label in the top-left corner which I'll refer to in this blog post. A larger version of this image can be found at http://buytaert.net/sites/buytaert.net/files/images/blog/digital-distrib....

Scenario 1: Digital distributors provide commercial value to publishers (A1 → A3/B3)

Digital distributors provide publishers reach, but without tangible commercial benefits, they risk being perceived as diluting or even destroying value for publishers rather than adding it. Right now, digital distributors are in early, experimental phases of enabling publishers to monetize their content. Facebook's Instant Articles currently lets publishers retain 100 percent of revenue from the ad inventory they sell. Flipboard, in efforts to stave off rivals like Apple News, has experimented with everything from publisher paywalls to native advertising as revenue models. Expect much more experimentation with different monetization models and dealmaking between the publishers and digital distributors.

If digital distributors like Facebook succeed in delivering substantial commercial value to the publisher they may fully embrace the distributor model and even divest their own websites' front-end, especially if the publishers could make the vast majority of their revenue from Facebook rather than from their own websites. I'd be interested to see someone model out a business case for that tipping point. I can imagine a future upstart media company either divesting its website completely or starting from scratch to serve content directly to distributors (and being profitable in the process). This would be unfortunate news for the Open Web and would mean that content management systems need to focus primarily on multi-channel publishing, and less on their own presentation layer.

As we have seen from other industries, decoupling production from consumption in the supply-chain can redefine industries. We also know that introduces major risks as it puts a lot of power and control in the hands of a few.

Scenario 2: The Open Web's disruptive innovation happens (A1 → C1/C2)

For the Open Web to win, the next disruptive innovation must focus on narrowing the usability gap with distributors. I've written about a concept called a Personal Information Broker (PIM) in a past post, which could serve as a way to responsibly use customer data to engineer similar personal, contextually relevant experiences on the Open Web. Think of this as unbundling Facebook where you separate the personal information management system from their content aggregation and curation platform, and make that available for everyone on the web to use. First, it would help us to close the user experience gap because you could broker your personal information with every website you visit, and every website could instantly provide you a contextual experience regardless of prior knowledge about you. Second, it would enable the creation of more distributors. I like the idea of a PIM making the era of handful of closed distributors as short as possible. In fact, it's hard to imagine the future of the web without some sort of PIM. In a future post, I'll explore in more detail why the web needs a PIM, and what it may look like.

Scenario 3: Coexistence (A1 → A2/B1/B2)

Finally, in a third combined scenario, neither publishers nor distributors dominate, and both continue to coexist. The Open Web serves as both a content hub for distributors, and successfully uses contextualization to improve the user experience on individual websites.

Conclusion

Right now, since distributors are out-innovating on relevance and discovery, publishers are somewhat at their mercy for traffic. However, a significant enough profit motive to divest websites completely remains to be seen. I can imagine that we'll continue in a coexistence phase for some time, since it's unreasonable to expect either the Open Web or digital distributors to fail. If we work on the next disruptive technology for the Open Web, it's possible that we can shift the pendulum in favor of “open” and narrow the usability gap that exists today. If I were to guess, I'd say that we'll see a move from A1 to B2 in the next 5 years, followed by a move from B2 to C2 over the next 5 to 10 years. Time will tell!

Acquia announces it is ready for Drupal 8

I'm excited to announce that starting today, Acquia is announcing we're ready to fully support our customers with Drupal 8. This means our professional services, our support, our product engineering, our cloud services … the entire company is ready to help anyone with Drupal 8 starting today.

While Drupal 8 is not yet released (as it has always been said, Drupal 8 will be "ready when it's ready"), the list of release blockers is dwindling ever closer to zero, and a beta-to-beta upgrade path will soon be provided in core. These factors, along with Acquia's amazing team of more than 150 Drupal experts (including a dedicated Drupal 8 engineering team that has contributed to fixing more than 1,200 Drupal 8 issues), gives us full confidence that we can make our customers successful with Drupal 8 starting today.

In the process of working with customers on their Drupal 8 projects, we will contribute Drupal 8 core patches, port modules, help improve Drupal 8's performance and more.

I'm excited about this milestone, as Drupal 8 will be a truly ground-breaking release. I'm most excited about the architectural enhancements that strongly position Drupal 8 for what I've called the Big reverse of the Web. For the web to reach its full potential, it will go through a massive re-platforming. From Flipboard to the upcoming release of Apple News, it's clear that the web is advancing into the “post-browser” era, where more and more content is "pushed" to you by smart aggregators. In this world, the traditional end-point of the browser and website become less relevant, requiring a new approach that increases the importance of structured content, metadata and advanced caching. With Drupal 8, we've built an API-driven architecture that is well suited to this new “content as a service” approach, and Drupal 8 is ahead of competitive offerings that still treat content as pages. Check out my DrupalCon Los Angeles keynote for more details.

One year later: the Acquia Certification Program

A little over a year ago we launched the Acquia Certification Program for Drupal. We ended up the first year with close to 1,000 exams taken, which exceeded our goal of 300-600. Today, I'm pleased to announce that the Acquia Certification Program passed another major milestone with over 1,000 exams passed (not just taken).

People have debated the pros and cons of software certifications for years (including myself) so I want to give an update on our certification program and some of the lessons learned.

Acquia's certification program has been a big success. A lot of Drupal users require Acquia Certification; from the Australian government to Johnson & Johnson. We also see many of our agency partners use the program as a tool in the hiring process. While a certification exam can not guarantee someone will be great at their job (e.g. we only test for technical expertise, not for attitude), it does give a frame of reference to work from. The feedback we have heard time and again is how the Acquia Certification Program is tough, but fair; validating skills and knowledge that are important to both customers and partners.

We also made the Certification Magazine Salary Survey as having one of the most desired credentials to obtain. To be a first year program identified among certification leaders like Cisco and Red Hat speaks volumes on the respect our program has established.

Creating a global certification program is resource intensive. We've learned that it requires the commitment of a team of Drupal experts to work on each and every exam. We now have four different exams: developer, front-end specialist, backend specialist and site builder. It roughly takes 40 work days for the initial development of one exam, and about 12 to 18 work days for each exam update. We update all four of our exams several times per year. In addition to creating and maintaining the certification programs, there is also the day-to-day operations for running the program, which includes providing support to participants and ensuring the exams are in place for testing around the globe, both on-line and at test centers. However, we believe that effort is worth it, given the overall positive effect on our community.

We also learned that benefits are an important part to participants and that we need to raise the profile of someone who achieves these credentials, especially those with the new Acquia Certified Grand Master credential (those who passed all three developer exams). We have a special Grand Master Registry and look to create a platform for these Grand Masters to help share their expertise and thoughts. We do believe that if you have a Grand Master working on a project, you have a tremendous asset working in your favor.

At DrupalCon LA, the Acquia Certification Program offered a test center at the event, and we ended up having 12 new Grand Masters by the end of the conference. We saw several companies stepping up to challenge their best people to achieve Grand Master status. We plan to offer the testing at DrupalCon Barcelona, so take advantage of the convenience of the on-site test center and the opportunity to meet and talk with Peter Manijak, who developed and leads our certification efforts, myself and an Acquia Certified Grand Master or two about Acquia Certification and how it can help you in your career!

Winning back the Open Web

The web was born as an open, decentralized platform allowing different people in the world to access and share information. I got online in the mid-nineties when there were maybe 100,000 websites in the world. Google didn't exist yet and Steve Jobs had not yet returned to Apple. I remember the web as an "Open Web" where no one was really in control and everyone was able to participate in building it. Fast forward twenty years, and the web has taken the world by storm. We now have a hundreds of millions of websites. Look beyond the numbers and we see another shift: the rise of a handful of corporate "Walled Gardens" like Facebook, Google and Apple that are becoming both the entry point and the gatekeepers of the web. Their dominance has given rise to major concerns.

We call them "Walled Gardens" because they control the applications, content and media on their platform. Examples include Facebook or Google, which control what content we get to see; or Apple, which restricts us to running approved applications on iOS. This is in contrast to the Open Web, where users have unrestricted access to applications, content and media.

Facebook is feeling the heat from Google, Google is feeling the heat from Apple but none of these Walled Gardens seem to be feeling the heat from an Open Web that safeguards our privacy and our society's free flow of information.

This blog post is the result of people asking questions and expressing concerns about a few of my last blog posts like the Big Reverse of the Web, the post-browser era of the web is coming and my DrupalCon Los Angeles keynote. Questions like: Are Walled Gardens good or bad? Why are the Walled Gardens winning? And most importantly; how can the Open Web win? In this blog post, I'd like to continue those conversations and touch upon these questions.

Are Walled Gardens good or bad for the web?

What makes this question difficult is that the Walled Gardens don't violate the promise of the web. In fact, we can credit them for amplifying the promise of the web. They have brought hundreds of millions of users online and enabled them to communicate and collaborate much more effectively. Google, Apple, Facebook and Twitter have a powerful democratizing effect by providing a forum for people to share information and collaborate; they have made a big impact on human rights and civil liberties. They should be applauded for that.

At the same time, their dominance is not without concerns. With over 1 billion users each, Google and Facebook are the platforms that the majority of people use to find their news and information. Apple has half a billion active iOS devices and is working hard to launch applications that keep users inside their walled garden. The two major concerns here are (1) control and (2) privacy.

First, there is the concern about control, especially at their scale. These organizations shape the news that most of the world sees. When too few organizations control the media and flow of information, we must be concerned. They are very secretive about their curation algorithms and have been criticized for inappropriate censoring of information.

Second, they record data about our behavior as we use their sites (and the sites their ad platforms serve) inferring information about our habits and personal characteristics, possibly including intimate details that we might prefer not to disclose. Every time Google, Facebook or Apple launch a new product or service, they are able to learn a bit more about everything we do and control a bit more about our life and the information we consume. They know more about us than any other organization in history before, and do not appear to be restricted by data protection laws. They won't stop until they know everything about us. If that makes you feel uncomfortable, it should. I hope that one day, the world will see this for what it is.

While the Walled Gardens have a positive and democratizing impact on the web, who is to say they'll always use our content and data responsibly? I'm sure that to most critical readers of this blog, the Open Web sounds much better. All things being equal, I'd prefer to use alternative technology that gives me precise control over what data is captured and how it is used.

Why are the Walled Gardens winning?

Why then are these Walled Gardens growing so fast? If the Open Web is theoretically better, why isn't it winning? These are important questions about future of the Open Web, open source software, web standards and more. It is important to think about how we got to a point of walled garden dominance, before we can figure out how an open web can win.

The biggest reason the Walled Gardens are winning is because they have a superior user experience, fueled by data and technical capabilities not easily available to their competitors (including the Open Web).

Unlike the Open Web, Walled Gardens collect data from users, often in exchange for free use of a service. For example, having access to our emails or calendars is incredibly important because it's where we plan and manage our lives. Controlling our smartphones (or any other connected devices such as cars or thermostats) provides not only location data, but also a view into our day-to-day lives. Here is a quick analysis of the types of data top walled gardens collect and what they are racing towards:

Walled gardens data

On top of our personal information, these companies own large data sets ranging from traffic information to stock market information to social network data. They also possess the cloud infrastructure and computing power that enables them to plow through massive amounts of data and bring context to the web. It's not surprising that the combination of content plus data plus computing power enables these companies to build better user experiences. They leverage their data and technology to turn “dumb experiences” into smart experiences. Most users prefer smart contextual experiences because they simplify or automate mundane tasks.

Walled gardens technology

Can the Open Web win?

I still believe in the promise of highly personalized, contextualized information delivered directly to individuals, because people ultimately want better, more convenient experiences. Walled Gardens have a big advantage in delivering such experiences, however I think the Open Web can build similar experiences. For the Open Web to win, we first must build websites and applications that exceed the user experience of Facebook, Apple, Google, etc. Second, we need to take back control of our data.

Take back control over the experience

The obvious way to build contextual experiences is by combining different systems that provide open APIs; e.g. we can integrate Drupal with a proprietary CRM and commerce platform to build smart shopping experiences. This is a positive because organizations can take control over the brand experience, the user experience and the information flow. At the same time users don't have to trust a single organization with all of our data.

Open web current state

The current state of the web: one end-user application made up of different platform that each have their own user experience and presentation layer and stores its own user data.

To deliver the best user experience, you want “loosely-coupled architectures with a highly integrated user experience”. Loosely-coupled architectures so you can build better user experiences by combining your systems of choice (e.g. integrate your favorite CMS with your favorite CRM with your favorite commerce platform). Highly-integrated user experiences so can build seamless experiences, not just for end-users but also for content creators and site builders. Today's Open Web is fragmented. Integrating two platforms often remains difficult and the user experience is "mostly disjointed" instead of "highly integrated". As our respective industries mature, we must focus our attention to integrating the user experience as well as the data that drives that user experience. The following "marketecture" illustrates that shift:

Shared integration and user experience layer

Instead of each platform having its own user experience, we have a shared integration and presentation layer. The central integration layer serves to unify data coming from distinctly different systems. Compatible with the "Big Reverse of the Web" theory, the presentation layers is not limited to a traditional web browser but could include push technology like a notification.

For the time being, we have to integrate with the big Walled Gardens. They need access to great content for their users. In return, they will send users to our sites. Content management platforms like Drupal have a big role to play, by pushing content to these platforms. This strategy may sound counterintuitive to many, since it fuels the growth of Walled Gardens. But we can't afford to ignore ecosystems where the majority of users are spending their time.

Control personal data

At the same time, we have to worry about how to leverage people's data while protecting their privacy. Today, each of these systems or components contain user data. The commerce system might have data about past purchasing behavior, the content management system about who is reading what. Combining all the information we have about a user, across all the different touch-points and siloed data sources will be a big challenge. Organizations typically don't want to share user data with each other, nor do users want their data to be shared without their consent.

The best solution would be to create a "personal information broker" controlled by the user. By moving the data away from the applications to the user, the user can control what application gets access to what data, and how and when their data is shared. Applications have to ask the user permission to access their data, and the user explicitly grants access to none, some or all of the data that is requested. An application only gets access to the data that we want to share. Permissions only need to be granted once but can be revoked or set to expire automatically. The application can also ask for additional permissions at any time; each time the person is asked first, and has the ability to opt out. When users can manage their own data and the relationships they have with different applications, and by extension with the applications' organizations, they take control over their own privacy. The government has a big role to play here; privacy law could help accelerate the adoption of "personal information brokers".

Open web personal information broker

Instead of each platform having its own user data, we move the data away from the applications to the users, managed by a "personal information broker" under the user's control.

Open web shared broker

The user's personal information broker manages data access to different applications.

Conclusion

People don't seem so concerned about their data being hosted with these Walled Gardens since they've willingly given it to date. For the time being, "free" and "convenient" will be hard to beat. However, my prediction is that these data privacy issues are going to come to a head in the next five to ten years, and lack of transparency will become unacceptable to people. The Open Web should focus on offering user experiences that exceed those provided by Walled Gardens, while giving users more control over their user data and privacy. When the Open Web wins through improved transparency, the closed platforms follow suit, at which point they'll no longer be closed platforms. The best case scenario is that we have it all: a better data-driven web experience that exists in service to people, not in the shadows.

The post-browser era of the web is coming

At yesterday's Worldwide Developer Conference keynote, Apple announced its annual updates to iOS, OS X, and the new watchOS. As usual, the Apple rumor blogs correctly predicted most of the important announcements weeks ago, but one important piece of news only leaked a few hours before the keynote: the launch of a new application called "News". Apple's News app press release noted: "News provides beautiful content from the world's greatest sources, personalized for you".

Apple basically cloned Flipboard to create News. Flipboard was once Apple's "App of the Year" in 2010, and it remains one of the most popular reading applications on iOS. This isn't the first time Apple has chosen to compete with its ecosystem of app developers. There is even a term for it, called "Sherlocking".

But forget about Apple's impact on Flipboard for a minute. The release of the News app signifies a more important shift in the evolution of the web, the web content management industry, and the publishing industry.

Impact on content management platforms

Why is Apple's News app a big deal for content management platforms? When you can read all the news you are interested in in News, you no longer have to visit websites for it. It's a big deal because there are half a billion active iOS devices and Apple will ship its News app to every single one of them. It will accelerate the fact that websites are becoming less relevant as an end-point destination.

Some of the other new iOS 9 features will add fuel to the fire. For example, Apple's search service Spotlight will also get an upgrade, allowing third-party services to work directly with Apple's search feature. Spotlight can now "deep link" to content inside of a website or application, further eliminating website or applications as end-points. You could search for a restaurant in Yelp directly from your home screen, and go straight to Yelp's result page without having to open the Yelp website or application. Add to that the Apple Watch which doesn't even ship with a web browser, and it's clear that Apple is about to accelerate the post-browser era of the web.

The secret to the News app is the new Apple News Format; rumored to be a RSS-like data feed with support for additional design elements like images, videos, custom fonts, and more. Apple uses these feeds to aggregate content from different news sources, uses machine learning to match the best content to a given user, and provides a clean, consistent look and feel for articles coming from the various news sources. That is the long way of saying that Apple decides what the best content is for you, and what the best format is to deliver it in. It is a profound change, but for most people this will actually be a superior user experience.

The release of Apple News is further proof that data-driven experiences will be the norm and of what I have been calling The Big Reverse of the Web. The fact that for the web to reach its full potential, it will go through a massive re-architecture from a pull-based architecture to a push-based architecture. After the Big Reverse of the Web is complete, content will find you, rather than you having to find content. Apple's News and Flipboard are examples of what such push-based experiences look like; they "push" relevant and interesting content to you rather than you having to "pull" the news from multiple sources yourself.

When content is "pushed" to you by smart aggregators, using a regular web browser doesn't make much sense. You benefit from a different kind of browser for the web. For content management platforms, it redefines the browser and websites as end-points; de-emphasizing the role of presentation while increasing the importance of structured content and metadata. Given Apple's massive install base, the launch of its News app will further accelerate the post-browser era of the web.

I don't know about your content management platform, but Drupal is ready for it. It was designed for a content-first mentality while many competitive content management systems continue to rely on a dated page-centric content model. It was also designed to be a content repository capable of outputting content in multiple formats to multiple end-points.

Impact on publishing industry

Forget the impact on Flipboard or on content management platforms, the impact on the publishing world will even be more significant. The risk for publishers is that they are being disintermediated as the distribution channel and that their brands become less useful. It marks a powerful transformation that could de-materialize and de-monetize much of the current web and publishing industry.

Because of Apple's massive installed base, Apple will now own a large part of the distribution channel and it will have an outsized influence on what hundreds of millions of users will read. If we've learned one thing in the short history of the Internet, it is that jumping over middlemen is a well-known recipe for success.

This doesn't mean that online news media have lost. Maybe it can actually save them? Apple could provide publishers large and small with an immense distribution channel by giving them the ability to reach every iOS user. Apple isn't alone with this vision, as Facebook recently rolled out an experiment with select publishers like Buzzfeed and the New York Times called Instant Articles.

In a "push economy" where a publisher's brand is devalued and news is selected by smart aggregators, the best content could win; not just the content that is associated with the most well-known publishing brands with the biggest marketing budgets. Publishers will be incentivized to create more high-quality content -- content that is highly customized to different target audiences, rather than generic content that appeals to large groups of people. Success will likely rely on Apple's ability to use data to match the right content to each user.

Conclusion

This isn't necessarily bad. In my opinion, the web isn't dead, it's just getting started. We're well into the post-PC era, and now Apple is helping to move consumers beyond the browser. It's hard to not be cautiously optimistic about the long-term implications of these developments.

Pages

Dries Buytaert is the original creator and project lead of Drupal and the co-founder and CTO of Acquia. He writes about Drupal, startups, business, photography and building the world we want to exist in.