You are here

Always be shippable

Drupal will soon be 15 years old, and 5 of that will be spent on building Drupal 8 -- a third of Drupal's life. We started work on Drupal early in 2011 and targeted December 1, 2012 as the original code freeze date. Now almost three years later, we still haven't released Drupal 8. While we are close to the release of Drupal 8, I'm sure many many of you are wondering why it took 3 years to stabilize. It is not like we didn't work hard or that we aren't smart people. Quite the contrary, the Drupal community has some of the most dedicated, hardest working and smartest people I know. Many spent evenings and weekends pushing to get Drupal 8 across the finish line. No one individual or group is to blame for the delay -- except maybe me as the project lead for not having learned fast enough from previous release cycles.

Trunk-based development

The past 15 years we used "trunk-based development"; we built new features in incremental steps, maintained in a single branch called "trunk". We'd receive the feature's first patch, commit it to our main branch, put it behind us, and move on to the next patch. Trunk-based development requires a lot of discipline and as a community we have mostly mastered this style of development. We invested heavily in our testing infrastructure and established a lot of processes. For all patches, we had various people reviewing the work to make sure it was solid. We also had different people figure out and review all the required follow-up work to complete the feature. The next steps are carefully planned and laid out for everyone to see in what we call "meta" issues. The idea of splitting one large feature into smaller tasks is not a bad idea; it helps to make things manageable for everyone involved.

Given all this rigor, how do you explain the delays then? The problem is that once these features and plans meet reality, they fall apart. Some features such as Drupal 8's configuration management system had to be rewritten multiple times based on our experience using it despite passing a rigorous review process. Other features such as our work on URL routing, entity base fields and Twig templating required much more follow-up work compared to what was initially estimated. It turns out that breaking up a large task into smaller ones requires a lot of knowledge and vision. It's often impossible to estimate the total impact of a larger feature on other subsystems, overall performance, etc. In other cases, the people working on the feature lacked time or interest to do the follow-up work, leaving it to others to complete. We should realize is that this is how things work in a complex world and not something we are likely to change.

The real problem is the fact that our main branch isn't kept in a shippable state. A lot of patches get committed that require follow-up work, and until that follow-up work is completed, we can't release a new version of Drupal. We can only release as fast as the slowest feature, and this is the key reason why the Drupal 8 release is delayed by years.

Shippable trunk based development

Trunk-based development; all development is done on a single main branch and as a result we can only release as fast as the slowest feature.

We need a better way of working -- one that conforms to the realities of the world we live in -- and we need to start using it the day Drupal 8.0.0 is released. Instead of ignoring reality and killing ourselves trying to meet unrealistic release goals, we need to change the process.

Feature branch workflow

The most important thing we have to do is keep our main branch in a shippable state. In an ideal world, each commit or merge into the main branch gives birth to a release candidate — it should be safe to release after each commit. This means we have to stop committing patches that put our main branch in an unshippable state.

While this can be achieved using a trunk-based workflow, a newer and better workflow called "feature branch workflows" has become popular. The idea is that (1) each new feature is developed in its own branch instead of the main branch and that (2) the main branch only contains shippable code.

Keeping the main branch shippable at all times enables us to do frequent date-based releases. If a specific feature takes too long, development can continue in the feature branch, and we can release without it. Or when we are uncertain about a feature's robustness or performance, rather than delaying the release, it will simply have to wait until the next release. The maintainers decide to merge in a feature branch based on objective and subjective criteria. Objectively, the test suite must pass, the git history must be clean, etc. Subjectively, the feature must deliver value to the users while maintaining desirable characteristics like consistency (code, API, UX), high performance, etc.

Shippable feature branching

Feature branching; each feature is developed in a dedicated branch. A feature branch is only merged into the main branch when it is "shippable". We no longer have to wait for the slowest feature before we can create a new release.

Date-based releases are widely adopted in the Open Source community (Ubuntu, OpenStack, Android) and are healthy for Open Source projects; they reduce the time it takes for a given feature to become available to the public. This encourages contribution and is in line with the "release early, release often" mantra. We agreed on the benefits and committed to date-based releases following 8.0.0, so this simply aligns the tooling to make it happen.

Feature branch workflows have challenges. Reviewing a feature branch late in its development cycle can be challenging. There is a lot of change and discussion already incorporated. When a feature does finally integrate into main, a lot of change hits all at once. This can be psychologically uncomfortable. In addition, this can be disruptive to the other feature branches in progress. There is no way to avoid this disruption - someone has to integrate first. Release managers minimize the disruption by prioritizing high priority or low disruption feature branches over others.

Here is a workflow that could give us the best of both worlds. We create a feature branch for each major feature and only core committers can commit to feature branches. A team working on a feature would work in a sandbox or submit patches like we do today. Instead of committing patches to the main branch, core committers would commit patches to the corresponding feature branch. This ensures that we maintain our code review process with smaller changes that might not be shippable in isolation. Once we believe a feature branch to be in a shippable state, and it has received sufficient testing, we merge the feature branch into the main branch. A merge like this wouldn't require detailed code review.

Feature branches are also not the silver bullet to all problems we encountered with the Drupal 8 release cycle. We should keep looking for improvements and build them into our workflows to make life easier for ourselves and those we are implementing Drupal for. More on those in future posts.

How Acquia is addressing the explosion of sites

I believe that the "digitalization" of the world is a "megatrend" that will continue for decades. On the one hand, organizations are shifting their businesses online, often inventing new ways to do business. On the other hand, customers are expecting a better and smarter user experience online.

This has led to two important sub-trends: (1) the number of sites an organization is creating and managing is growing at a rapid clip, (2) so is the underlying complexity of each website.

Forrester Research recently surveyed large enterprises about their website portfolio and found that on average they manage 268 properties across various channels. On top of that, each website is becoming more and more advanced. They evolved from simple HTML pages to dynamic websites to digital experience platforms that need to integrate with many other business systems. The combination of these two trends -- increasing number of sites and the growing complexity of each site -- poses real challenges to most organizations.

At Acquia, we are seeing this explosion of websites in the enterprise every day. Many organizations have different websites for different brands and products, want different websites for each country or region they operate in, or offer separate portals for their affiliates, dealers, agents or franchises. We're also seeing organizations, small and large, operate a large number of marketing campaign websites. These organizations aren't focused on scaling back their online properties but rather how best to manage them over time.

I outlined this trend and its challenges almost five years ago (see Acquia product strategy and vision) and most of it is still relevant today, if not more relevant. In this blog post, I want to give you an update and share some lessons learned.

Current situation

Most larger organizations run many different types of websites. It's not unusual for a small organization to have ten websites and for a large organization to have hundreds of websites. Some of Acquia's largest customers operate thousands of websites.

Acquia cloud site factory many sites

Most organizations struggle to manage their growing portfolio of digital properties. You'd be surprised how many organizations have more than 20 different content management systems in use. Often this means that different teams are responsible for them and that they are hosted on different hosting environments. It is expensive, creates unnecessary security risks, poses governance challenges, leads to brand inconsistency, makes it difficult to create a unified customer experience, and more. It costs large organizations millions of dollars a year.

Acquia cloud site factory many platforms

Drupal's unfair advantage

When managing many sites, Drupal has an unfair advantage in that it scales from simple to complex easily. That scalability, coupled with a vast ecosystem of modules, elevate Drupal from a single site point solution to a platform on which you can build almost any kind of site: a brand site, a corporate website, a customer support community, a commerce website, an intranet, etc. You name it.

This is in contrast to many of Drupal's competitors that are either point solutions (e.g. SharePoint is mainly used for intranets) or whose complexity and cost don't lend themselves to managing many sites (e.g. Adobe Experience Manager and Sitecore are expensive solutions for a quick marketing campaign site, while WordPress can be challenging for building complex websites). So the first thing people can do is to standardize on Drupal as a platform for all of their site needs.

Acquia cloud site factory many drupals

By standardizing on Drupal, organizations can simplify training, reduce maintenance costs, streamline security and optimize internal resources – all without sacrificing quality or requirements. Standardizing on Drupal certainly doesn't mean every single site needs to be on Drupal. Transitioning from 20 different systems to 3 still translates into dramatic cost savings.

The Acquia advantage

Once an organization decides to standardize on Drupal, the question is how best to manage all these sites? In 2013 we launched Acquia Cloud Site Factory (ACSF), a scalable enterprise-grade multi-site management platform that helps organizations to easily create, deploy and govern all their sites. Today, some of Acquia's biggest customers use ACSF to manage hundreds of sites - in fact on average an ACSF customer is currently managing 170 websites within their Site Factory platform and that number is growing rapidly.

Acquia commissioned Forrester Research to analyze the benefits to organizations who have unified their sites on a single platform. Forrester found that moving to a single platform dramatically reduced site development and support costs, conserved IT and marketing resources, and improved standardization, governance and scalability — all while accelerating time-to-market and the delivery of better digital experiences.

One of the things we've learned is that a complete multi-site management solution needs to include advanced tools for both developers and content managers. The following image illustrates the different layers of a complete multi-site management solution:

Acquia Cloud Site Factory solution stack

The different layers of the Acquia Cloud Site Factory solution stack.

Let's go through these individually from the bottom up.

Infrastructure management

Consider an organization that currently has 50 websites, and plans to add 10-15 more sites every year. With ACSF these sites run on a platform that is scalable, secure and highly reliable. This infrastructure also allows hardware resources to be logically isolated based on the site's needs as well as scaled up or down to meet any ad-hoc traffic spikes. These capabilities enable organizations to simplify multi-site management efforts and eliminate operational headaches.

Code management

If this organization with 50 sites had individual codebases for each site, that would be 50 disparate codebases to manage. With ACSF, the underlying code can be shared and managed in one central place but the content, configuration, and creative look-and-feel can be catered to each individual sites' needs. ACSF also enable developers to easily add or remove features from their codebases for individual sites. ACSF also comes with tools to automate the process of rolling out updates across all their sites.

Site management

Organizations with many sites also need efficient ways to manage and govern them effectively; from developer tools such as Git, Travis, or Behat that enable them to build, test and maintain sites, to tools for non-developers to quickly clone and spin up sites using site templates defined by a brand manager or a digital design team. ACSF enables customers to effortlessly manage all their sites from a single intuitive dashboard. Developers can create groups of users as well as sites allowing certain users to manage their dedicated domain of sites without stepping over other sites. Non-technical content managers can quickly spin up new sites by cloning existing ones they have access to and updating their configuration, content and look-and-feel. These features allow organizations to launch sites at unprecedented speed inherently improving their overall time to market.

Content sharing

Write once, publish anywhere. We learned from customers managing multiple sites that one thing they often need is the ability to easily share content between sites. For example, if an organization has a privacy policy that needs to be updated, it doesn't make sense to update all their 50 sites individually. There needs to be an easier way to discover existing content that can be repurposed across other sites as well as the ability to author new content once within a platform and deliver it to other sites as needed.

Personalization

Finally, I should mention personalization. For a few years now we have been developing Acquia Lift. Acquia Lift builds unified customer profiles across all your websites, and uses that information to deliver real-time, contextual, and personalized experiences. For instance, if the organization in the above example had 50 websites for each of their 50 different products, Acquia Lift can present relevant content to its users as they browse across these different sites. This enables organizations to convert anonymous site visitors into known customers and establish a meaningful engagement between them.

Conclusion

I believe that the "multi-sites era" will continue to accelerate; not only will we see more sites, but every site will become increasingly complex. Organizations need to think about how to efficiently manage their website portfolio. If you're not thinking ahead, you're falling behind.

Drupal 8 now ships with beta-to-beta updates

We've just achieved a big and exciting milestone in Drupal 8's development: starting with Drupal 8 beta 15, we are providing beta-to-beta upgrade paths. This will make it much easier to update Drupal 8 development sites between the current beta and future betas and release candidates.

There has been a lot of excitement building around Drupal 8. Many have been wondering when to start building Drupal 8 sites. The answer for many is NOW.

This change signals an important opportunity for organizations to begin developing with Drupal 8, especially for:

  • Sites that rely mainly on the expanded functionality provided by Drupal 8 core alone.
  • Projects that will take months of development time.
  • Sites for which the benefits of Drupal 8's outweigh the effort needed to port (or work around) contributed modules that do not yet have Drupal 8 versions.

I strongly encourage you to evaluate Drupal 8 for your upcoming projects. Also, if you haven't already, now is the time to port contributed modules so they are ready in time for Drupal 8's release! There are only about five release-blocking issues left before we create the first release candidate.

Note that betas are not supported releases of Drupal, and both developing and launching sites with beta releases present risks. However, I'm pleased that various Drupal agencies, including Acquia, are helping to eliminate those risks through support, development, and hosting optimized for Drupal 8.

Before you get started with Drupal 8, be sure to review all the release notes for beta 15.

Digital Distributors vs Open Web: who will win?

I've spent a fair amount of time thinking about how to win back the Open Web, but in the case of digital distributors (e.g. closed aggregators like Facebook, Google, Apple, Amazon, Flipboard) superior, push-based user experiences have won the hearts and minds of end users, and enabled them to attract and retain audience in ways that individual publishers on the Open Web currently can't.

In today's world, there is a clear role for both digital distributors and Open Web publishers. Each needs the other to thrive. The Open Web provides distributors content to aggregate, curate and deliver to its users, and distributors provide the Open Web reach in return. The user benefits from this symbiosis, because it's easier to discover relevant content.

As I see it, there are two important observations. First, digital distributors have out-innovated the Open Web in terms of conveniently delivering relevant content; the usability gap between these closed distributors and the Open Web is wide, and won't be overcome without a new disruptive technology. Second, the digital distributors haven't provided the pure profit motives for individual publishers to divest their websites and fully embrace distributors.

However, it begs some interesting questions for the future of the web. What does the rise of digital distributors mean for the Open Web? If distributors become successful in enabling publishers to monetize their content, is there a point at which distributors create enough value for publishers to stop having their own websites? If distributors are capturing market share because of a superior user experience, is there a future technology that could disrupt them? And the ultimate question: who will win, digital distributors or the Open Web?

I see three distinct scenarios that could play out over the next few years, which I'll explore in this post.

Digital Distributors vs Open Web: who will win?

This image summarizes different scenarios for the future of the web. Each scenario has a label in the top-left corner which I'll refer to in this blog post. A larger version of this image can be found at http://buytaert.net/sites/buytaert.net/files/images/blog/digital-distrib....

Scenario 1: Digital distributors provide commercial value to publishers (A1 → A3/B3)

Digital distributors provide publishers reach, but without tangible commercial benefits, they risk being perceived as diluting or even destroying value for publishers rather than adding it. Right now, digital distributors are in early, experimental phases of enabling publishers to monetize their content. Facebook's Instant Articles currently lets publishers retain 100 percent of revenue from the ad inventory they sell. Flipboard, in efforts to stave off rivals like Apple News, has experimented with everything from publisher paywalls to native advertising as revenue models. Expect much more experimentation with different monetization models and dealmaking between the publishers and digital distributors.

If digital distributors like Facebook succeed in delivering substantial commercial value to the publisher they may fully embrace the distributor model and even divest their own websites' front-end, especially if the publishers could make the vast majority of their revenue from Facebook rather than from their own websites. I'd be interested to see someone model out a business case for that tipping point. I can imagine a future upstart media company either divesting its website completely or starting from scratch to serve content directly to distributors (and being profitable in the process). This would be unfortunate news for the Open Web and would mean that content management systems need to focus primarily on multi-channel publishing, and less on their own presentation layer.

As we have seen from other industries, decoupling production from consumption in the supply-chain can redefine industries. We also know that introduces major risks as it puts a lot of power and control in the hands of a few.

Scenario 2: The Open Web's disruptive innovation happens (A1 → C1/C2)

For the Open Web to win, the next disruptive innovation must focus on narrowing the usability gap with distributors. I've written about a concept called a Personal Information Broker (PIM) in a past post, which could serve as a way to responsibly use customer data to engineer similar personal, contextually relevant experiences on the Open Web. Think of this as unbundling Facebook where you separate the personal information management system from their content aggregation and curation platform, and make that available for everyone on the web to use. First, it would help us to close the user experience gap because you could broker your personal information with every website you visit, and every website could instantly provide you a contextual experience regardless of prior knowledge about you. Second, it would enable the creation of more distributors. I like the idea of a PIM making the era of handful of closed distributors as short as possible. In fact, it's hard to imagine the future of the web without some sort of PIM. In a future post, I'll explore in more detail why the web needs a PIM, and what it may look like.

Scenario 3: Coexistence (A1 → A2/B1/B2)

Finally, in a third combined scenario, neither publishers nor distributors dominate, and both continue to coexist. The Open Web serves as both a content hub for distributors, and successfully uses contextualization to improve the user experience on individual websites.

Conclusion

Right now, since distributors are out-innovating on relevance and discovery, publishers are somewhat at their mercy for traffic. However, a significant enough profit motive to divest websites completely remains to be seen. I can imagine that we'll continue in a coexistence phase for some time, since it's unreasonable to expect either the Open Web or digital distributors to fail. If we work on the next disruptive technology for the Open Web, it's possible that we can shift the pendulum in favor of “open” and narrow the usability gap that exists today. If I were to guess, I'd say that we'll see a move from A1 to B2 in the next 5 years, followed by a move from B2 to C2 over the next 5 to 10 years. Time will tell!

Acquia announces it is ready for Drupal 8

I'm excited to announce that starting today, Acquia is announcing we're ready to fully support our customers with Drupal 8. This means our professional services, our support, our product engineering, our cloud services … the entire company is ready to help anyone with Drupal 8 starting today.

While Drupal 8 is not yet released (as it has always been said, Drupal 8 will be "ready when it's ready"), the list of release blockers is dwindling ever closer to zero, and a beta-to-beta upgrade path will soon be provided in core. These factors, along with Acquia's amazing team of more than 150 Drupal experts (including a dedicated Drupal 8 engineering team that has contributed to fixing more than 1,200 Drupal 8 issues), gives us full confidence that we can make our customers successful with Drupal 8 starting today.

In the process of working with customers on their Drupal 8 projects, we will contribute Drupal 8 core patches, port modules, help improve Drupal 8's performance and more.

I'm excited about this milestone, as Drupal 8 will be a truly ground-breaking release. I'm most excited about the architectural enhancements that strongly position Drupal 8 for what I've called the Big reverse of the Web. For the web to reach its full potential, it will go through a massive re-platforming. From Flipboard to the upcoming release of Apple News, it's clear that the web is advancing into the “post-browser” era, where more and more content is "pushed" to you by smart aggregators. In this world, the traditional end-point of the browser and website become less relevant, requiring a new approach that increases the importance of structured content, metadata and advanced caching. With Drupal 8, we've built an API-driven architecture that is well suited to this new “content as a service” approach, and Drupal 8 is ahead of competitive offerings that still treat content as pages. Check out my DrupalCon Los Angeles keynote for more details.

Pages

Updates from Dries straight to your mailbox