You are here

Drupal

The power of self-managed teams in Drupal

The concept of official initiatives came out of lessons learned from the Drupal 7 development. We learned a lot from that and in a recent blog post about Drupal initiative leads, I recognized that we need to evolve our tools, our processes, and our organizational design. Others like Nathaniel Catchpole, Larry Garfield and Gábor Hojtsy have shared some of their thoughts already. One of the things I'm most proud of is that the Drupal community is always looking to improve and reinvent itself. Evolving is an important part of our culture. Each time it will get better, but still won't be perfect.

For me, one of the biggest take-aways (but not the only one) is that for an initiative to succeed, it needs to be supported by a team. An initiative needs to carry out a technical vision, plan the work, communicate with all stakeholders, mobilize volunteers, raise funding, organize sprints, and more. It can easily be more than one person can handle -- especially if it isn't your full-time job or if your initiative is complex.

More specifically, we have learned that the most successful initiatives appear to be run by teams that are self-managed; the team members collaborate in the development of the initiative, but also share both managerial and operational responsibilities like planning, coordinating, communicating, sprint organizing and more.

Because self-managed teams are both responsible for their outcomes and in control of their decision-making process, members of a self-managing team are usually more motivated than traditional hierarchical teams. This independence and greater responsibility are important in volunteer communities. Self-managed teams also build and maintain institutional knowledge. The outcome of their work is also more easily accepted by other stakeholders (like core committers) because they have already built a lot of consensus.

If I were to be an initiative lead, I'd feel strongly about building my own team rather than being handed a team. My initial assumption was that each initiative lead would build his/her own team. In hindsight, that was a mistake. Team building is not easy. It requires a time investment that can seem to compete with technical priorities. This is an important lesson and something we can do better going forward. Before making an initiative official, we have to make sure that each initiative has a good team and the support to be successful -- either we can help create a team, provide more coaching or formal training around team building, or we shouldn't designate the initiative official until such a team has coalesced.

Weather.com using Drupal

One of the world's most trafficked websites, with more than 100 million unique visitors every month and more than 20 million different pages of content, is now using Drupal. Weather.com is a top 20 U.S. site according to comScore. As far as I know, this is currently the biggest Drupal site in the world.

Weather.com has been an active Drupal user for the past 18 months; it started with a content creation workflow on Drupal to help its editorial team publish content to its existing website faster. With Drupal, Weather.com was able to dramatically reduce the number of steps that was required to publish content from 14 to just a few. Speed is essential in reporting the weather, and Drupal's content workflow provided much-needed velocity. The success of that initial project is what led to this week's migration of Weather.com from Percussion to Drupal.

The company has moved the entire website to Acquia Cloud, giving the site a resilient platform that can withstand sudden onslaughts of demand as unpredictable as the weather itself. As we learned from our work with New York City's MTA during Superstorm Sandy in 2012, “weather-proofing” the delivery of critical information to insure the public stays informed during catastrophic events is really important and can help save lives.

The team at Weather.com worked with Acquia and Mediacurrent for its site development and migration.

Weather channel

The job of Drupal initiative lead

Drupal 8 is the first time we introduced the concept of formal initiatives and initiative leads. Over the course of these Drupal 8 initiatives we learned a lot and people are floating several ideas to increase the initiatives' success and provide Drupal initiative leads with more support. As we grow, it is crucial that we evolve our tools, our processes, and our organizational design based on these learnings. We've done so in the past and we'll continue to do so going forward.

But let's be honest, no matter how much support we provide, leading a Drupal initiative will unquestionably remain difficult and overwhelming. As a Drupal initiative lead, you are asked to push forward some of the most difficult and important parts of Drupal.

You will only succeed if you are able to build a strong team of volunteers that is willing to be led by you. You have to learn how to inspire and motivate by articulating a vision. You establish credibility by setting clear objectives and roadmaps in partnership with others. You have to motivate, guide and empower people to participate. You have to plan and over-communicate.

Not only do you have to worry about building and leading a team, you also have to make sure the rest of the community has shared goals and that everyone impacted has a shared understanding of why those decisions are being made. You use data, ideas and feedback from different sources to inform and convince people of your direction. Your "soft skills" are more important than your "hard skills". Regardless, you will lose many battles. You only "win" when you remain open to feedback and value change and collaboration. To lead a community, you need both a thick skin and a big heart.

Success is never a coincidence. You put in long hours to try and keep your initiative on track. You need relentless focus on doing whatever is necessary to succeed; to be the person who fills all the gaps and helps others to be successful. Instead of just doing the things you love doing most, you find yourself doing mundane tasks like updating spreadsheets or planning a code sprint to help others be successful. In fact, you might need to raise money for your code sprint. And if you succeed, you still don't have enough money to achieve what is possible and you feel the need to raise even more. You'll be brushing aside or knocking down obstacles in your path, and taking on jobs and responsibilities you have never experienced before.

Your objectives will constantly shift as Drupal itself iterates and evolves. You will want to go faster and you will struggle with the community processes. Imagine working on something for a month and then having to throw it out completely because you realize it doesn't pass. Frustration levels will be off the charts. Your overall goal of achieving the perfect implementation might never be achieved and that feeling haunts you for weeks or months. You will feel the need to vent publicly, and you probably will. At the worst moments, you'll think about stepping down. In better times, you realize that if most of your initiative succeeds it could take years of follow-up work. You will learn a lot about yourself; you learn that you are bad at many things and really good at other things.

Leading is incredibly hard and yet, it will be one of the best thing you ever did. You work with some of the finest, brightest, and most passionate people in the world. You will see tangible results of your hard work and you will impact and help hundreds of thousands of people for the next decade. There is no better feeling than when you inspire or when you help others succeed. Leading is hard, but many of you will look back at your time and say this was the most gratifying thing you ever did. You will be incredibly proud of yourself, and the community will be incredibly proud of you. You will become a better leader, and that will serve you for the rest of your life.

Acquia a leader in Gartner Magic Quadrant for Web Content Management

You might have read that Acquia was named a leader in the Gartner Magic Quadrant for Web Content Management.

It's easy to underestimate the importance of this recognition for Acquia, and by extension for Drupal. If you want to find a good coffee place, you use Yelp. If you want to find a nice hotel in New York, you use TripAdvisor. Similarly, if a CIO wants to spend $250,000 or more on enterprise software, they consult an analyst firm like Gartner. So think of Gartner as "Yelp for the enterprise".

Many companies create their technology shortlist based on the leader quadrant. That means that Drupal has not been considered as an option for hundreds of evaluations for large projects that have taken place in the past couple of years. Being named a leader alongside companies like Adobe, HP, IBM, Oracle, and Sitecore will encourage more organizations to evaluate Drupal. More organizations evaluating Drupal should benefit the Drupal ecosystem and the development of Drupal.

State of Drupal presentation (September 2014)

I gave my traditional State of Drupal presentation this week at DrupalCon Amsterdam. I decided to talk about the sustainability and scalability of the Drupal community. In case you didn't attend DrupalCon Amsterdam, you can watch the recording of my keynote, download a copy of my slides (PDF, 17 MB) or read my blog post on the topic.

Drupal 8 beta 1 released

Today we announced Drupal 8 beta 1! This key milestone is the work of over 2,300 people who have contributed more than 11,500 committed patches to 15 alpha releases, and especially the 234 contributors who fixed 178 "beta blocker" issues. A massive thank-you to everyone who helped get Drupal 8 beta 1 done.

For more information on the beta, please read the beta 1 release announcement. To read about the new features in Drupal 8, see Drupal.org's Drupal 8 landing page.

Betas are for developers and site builders who are comfortable reporting (and where possible, fixing) their own bugs, and who are prepared to rebuild their test sites from scratch if necessary. Beta releases are not recommended for non-technical users, nor for production websites.

Scaling Open Source communities

We truly live in miraculous times. Open Source is at the core of the largest organizations in the world. Open Source is changing lives in emerging countries. Open Source has changed the tide of governments around the world. And yet, Open Source can be really difficult. Open Source can be largely a thankless job. It is hard to find volunteers, it is hard to find organizations to donate time or money, it is hard to organize the community, it is hard to learn, it is hard to attract full-time contributors, and more. As the project lead for Drupal, one of the largest Open Source projects/communities in the world, I live these challenges every day. In this blog post, I will analyze the challenge with scaling Open Source communities and recommend a solution for how to build very large Open Source communities.

Open Source projects are public goods

In economic terms, for something to be a "public good", it needs to match two criteria:

  1. non-excludability - it is impossible to prevent anyone from consuming that good, and
  2. non-rivalry - consumption of this good by anyone does not reduce the benefits available to others.

Examples of public goods include street lighting, national defense, public parks, basic education, the road system, etc. By that definition, Open Source software is also a "public good": we can't stop anyone from using Open Source software, and one person benefiting from Open Source software does not reduce the benefits available to others.

The realization that Open Source is a public good is a helpful one because there has been a lot of research about how to maintain and scale public goods.

Public goods and the free-rider problem

The biggest problem with public goods is the "free rider problem". A free rider is someone who uses a public good but who does not pay anything (or pay enough) towards its cost or production. If the maintainers of a public good do not address the free-rider problem it can lead to the non-production or under-production of a public good. This is generally known as the "Tragedy of the Commons".

In Open Source, a free-rider is someone who uses an Open Source software project without contributing to it. If too few people or organizations contribute to the project, the project can become unhealthy, and ultimately could cease to exist.

The free-rider problem is typical for public goods and does not usually arise with private businesses. For example, community-maintained software like Drupal may have many free riders but proprietary competitors like Adobe or Sitecore have no problem excluding those who will not pay a license fee.

To properly understand the free-rider problem and public good provision, we need to understand both self-interest theory and the theory of collective action. I'll discuss both theories and apply them to Open Source.

Self-interest theory

Open Source contributors do amazing things. They contribute to fixing the hardest problems, they share their expertise, and more. Actions like these are often described as altruistic, in contrast to the pursuit of self-interest. In reality, generosity is often driven by some level of self-interest: we provide value to others when it benefits ourselves.

Many reasons exist why people contribute to Open Source projects; people contribute because they enjoy being part of a community of like-minded people, to hone their technical skills, to get recognition, to try and make a difference in the world, because they are paid to, or for different forms of "social capital". Often we contribute because by improving the world we are living in, we are making our world better too.

Modern economics suggest that both individuals and organizations tend to act in their own self-interest, bound by morals, ethics, the well-being of future generations and more. The theory of self-interest goes back to the writings of the old Greeks, is championed by early modern economists, and is still being adhered to by late-modern economists. It follows from the theory of self-interest that we'd see more individuals and organizations contribute if they received more benefits.

While contributing to Open Source clearly has benefits, it is not obvious if the benefits outweigh the cost. If we can increase the benefits, there is no doubt we can can attract more contributors.

Collective action theory

The theory of self-interest also applies to groups of individuals. In his seminal work on collective action and public goods, economist Mancur Olson shows that the incentive for group action diminishes as group size increases. Large groups are less able to act in their common interest than small ones because (1) the complexity increases and (2) the benefits diminish.

We see this first hand in Open Source projects. As an Open Source project grows, aspects of the development, maintenance and operation have to be transferred from volunteers to paid workers. Linux is a good example. Without Red Hat, IBM and Dell employing full-time Linux contributors, Linux might not have the strong market share it has today.

The concept of major public goods growing out of volunteer and community-based models is not new to the world. The first trade routes were ancient trackways, which citizens later developed on their own into roads suited for wheeled vehicles in order to improve commerce. Transportation was improved for all citizens, driven by the commercial interest of some. Today, we certainly appreciate that full-time government workers maintain the roads. Ditto for the national defense system, basic education, etc.

The theory of collective action also implies that as an Open Source project grows, we need to evolve how we incent contributors or we won't be able to attract either part-time volunteers or full-time paid contributors.

Selective benefits

Solutions for the free-rider problem and collective action problem exist, and this is where Open Source can learn from public goods theory and research. The most common solution for the free-rider problem is taxation; the government mandates all citizens to help pay for the production of the public good. Taxpayers help pay for our basic education system, the road system and national defense for example. Other solutions are privatization, civic duty or legislation. These solutions don't apply to Open Source.

I believe the most promising solution for Open Source is known as "privileged groups". Privileged groups are those who receive "selective benefits". Selective benefits are benefits that can motivate participation because they are available only to those who participate. The study of collective action shows that public goods are still produced when a privileged group benefits more from the public good than it costs them to produce it.

In fact, prominent "privileged groups" examples exist in the Open Source community; Automattic is a privileged group in the WordPress community as it is in a unique position to make many millions of dollars from WordPress.com. Mozilla Corporation, the for-profit subsidiary of the Mozilla Foundation, is a privileged group as it is in a unique position to get paid millions of dollars by Google. As a result, both Automattic and Mozilla Corporation are willing to make significant engineering investments in WordPress and Mozilla, respectively. Millions of people in the world benefit from that every day.

Drupal is different from Automattic and Mozilla in that no single organization benefits uniquely from contributing. For example, my company Acquia currently employs the most full-time contributors to Drupal but does not receive any exclusive benefits in terms of monetizing Drupal. While Acquia does accrue some value from hiring the Drupal contributors that it does, this is something any company can do.

Better incentives for Drupal contributors

It's my belief that we should embrace the concept of "privileged groups" and "selective benefits" in the Drupal community to help us grow and maintain the Drupal project. Furthermore, I believe we should provide "selective benefits" in a way that encourages fairness and equality, and doesn't primarily benefit any one particular organization.

From the theory of self-interest it follows that to get more paid core contributors we need to provide more and better benefits to organizations that are willing to let their employees contribute. Drupal agencies are looking for two things: customers and Drupal talent.

Many organizations would be eager to contribute more if, in return, they were able to attract more customers and/or Drupal talent. Hence, the "selective benefits" that we can provide them are things like:

  • Organizational profile pages on drupal.org with badges or statistics that prominently showcase their contributions,
  • Advertising on the drupal.org in exchange for fixing critical bugs in Drupal 8 (imagine we rewarded each company that helped fix a critical Drupal 8 bug 10,000 ad views on the front page of drupal.org),
  • Better visibility on Drupal.org's job board for those trying to hire Drupal developers,
  • The ability to sort the marketplace by contributions, rather than just alphabetically
  • ...

I'm particularly excited about providing ads in exchange for contributing. Contributing to Drupal now becomes a marketing expense; the more you contribute, the more customers you can gain from drupal.org. We can even direct resources; award more ad views in exchange for fixing UX problems early in the development cycle, but award critical bugs and beta blockers later in the development cycle. With some relatively small changes to drupal.org, hiring a full-time core developer becomes a lot more interesting.

By matching the benefits to the needs of Drupal agencies, we candirect more resources towards Drupal development. I also believe this system to be fair; all companies can choose to contribute to Drupal 8 and earn advertising credits, and all participants are rewarded equally. We can turn Drupal.org into a platform that encourages and directs participation from a large number of organizations.

Systems like this are subject to gaming but I believe these challenges can be overcome. Any benefit is better than almost no benefit. In general, it will be interesting to see if fairness and heterogeneity will facilitate or impede contribution compared to Open Source projects like WordPress and Mozilla, where some hold unique benefits. I believe that if all participants benefit equally from their contributions, they have an incentive to match each other's contributions and it will facilitate the agreement and establishment of a contribution norm that fosters both cooperation and coordination, while minimizing gaming of the system. In contrast, when participants benefit very differently, like with WordPress and Mozilla, this decreases the willingness to cooperate, which, in turn, could have detrimental effects on contributions. While not necessarily the easiest path, I believe that making the system fair and heterogeneous is the "Drupal way" and that it will serve us in the long term.

Conclusions

There are plenty of technical challenges ahead of us that we need to work on, fun ideas that we should experiment with, and more. With some relatively small changes, we could drastically change the benefits of contributing to Drupal. Better incentives mean more contributors, and more contributors mean that we can try more things and do things better and faster. It means we can scale Drupal development to new heights and with that, increase Open Source's impact on the world.

(I talked about this in my DrupalCon Amsterdam keynote. If you're hungry for more, I recommend that you check out my slides.)

Pages

© 1999-2014 Dries Buytaert Creative Commons Attribution-NonCommercial-ShareAlike 3.0 License.
Drupal is a Registered Trademark of Dries Buytaert.