Acquia works with many large enterprises that bet on Drupal. These organizations are doing amazing things with Drupal and innovating by breaking through prior limitations. However, in talking to our customers, we noticed that there is limited knowledge sharing and discussion happening among the heaviest Drupal users. Similar problems are often solved multiple times independently, and in incompatible ways. And since few of these large companies are vocal and active in the community, the expertise gained from solving these problems isn't making its way back into the software that all Drupal users rely on.
To help solve these issues, I'm announcing a new program called "Large Scale Drupal" as part of my group at Acquia's Office of the CTO.
Large Scale Drupal is a group of large enterprise Drupal users who meet regularly to discuss and collaborate on common problems. We provide a forum for enterprise users, listen to their needs, prioritize them as a group, and then figure out a proper way to address those needs through knowledge sharing, white papers, training and development. The intent is not to keep the outcome of these meetings just within the group. We want to share what we learn in the Large Scale Drupal group with the specific intent of it becoming a contributed project to Drupal. Once contributed, anyone is welcome to discuss and assist to the project.
So what are these projects? These are common needs for large enterprises that are considered large and complicated Drupal problems. Through a consensus-driven process our first project is working on creating a better content staging system geared toward supporting a publishing workflow. We've already started having detailed discussions and working on some of the basic architecture. We are connecting Large Scale Drupal program participants with members of the community to help advance projects like Workbench, and build new contributions like a site preview system. This program will add to those systems by helping define the needs of the users, funding some of the work, and contributing patches to the code.
The goal of these projects is to foster knowledge sharing and collaboration among members of the group and the community. The Large Scale Drupal members get the benefits of sharing their development costs with other members. The community benefits by gaining new contributions to Drupal, and an influx of expert talent into the Drupal contributor pool. Both the contributions of these companies, and the expertise that they bring to the table will help Drupal remain a long-term viable project.
I'm excited to work with the Large Scale Drupal program members to get them more involved in Drupal and become active contributors to the community. I have a big vision for Large Scale Drupal; something I hope to write more about later. For now, I wanted to announce and bring awareness to the program.
We've already made a number of process improvements to help accelerate Drupal core's development velocity. Today, I'd like to talk about how we can better streamline the Drupal core decision-making process.
The Drupal core queue is filled with extremely passionate, intelligent and determined individuals who always strive to solve problems in the best possible way. However, we do sometimes come to disagreements about the best way to approach a particular solution, and when two or more individuals fundamentally disagree, issues can sometimes turn into stalemates; or worse, turn ugly. When this happens, it saps strength from the core development team, and makes it difficult (and scary) for new contributors to engage.
Going forward, I'd like to propose the following workflow change to help un-stick issues where opinions seem deadlocked:
- Create a balanced issue summary at the top of the issue that reflects the current state and history of the discussion, and outlines the pros and cons of various options.
- Move the "Assigned to" field to "Dries". (Note that access to do this is restricted to those contributors listed in MAINTAINERS.txt; you can either ask one of them to do so on your behalf.)
- Once per week (barring travel or other things that might come up), I will go through this list and either make a decision myself, or delegate the decision to another contributor, along with a date for a decision to be made. If there is still not agreement by that date, the person delegated the responsibility will make what they feel is the best decision, and we'll move forward. If it's particularly contentious, we can tag it "revisit before release" and look at it again in November or so.
I hope that this process change will allow us to continue to make bold strides in Drupal 8, while giving dissenting opinions a chance to be heard.
It's that time of year again where we are gearing up for another great DrupalCon. Next week, 3000 Drupalists, including more than 70 Acquians, will be migrating out west to the Rocky Mountains for an action packed week filled with sessions, stickers, beer, and lots of face time with the best open source community on the planet.
There is one remarkable event that caught my attention and that speaks volumes about an important trend we're seeing: the Higher Ed Drupal Users meeting on Wednesday.
Why is this so interesting you ask? Well ... it all started with one of Acquia's team members reaching out to a couple of universities from Canada to organize a meeting at DrupalCon where they could connect and share insider tips for what works for them at their respective university. However, it turned out they were already talking on a regular basis and what they really wanted was to talk to others from universities outside of their immediate circle. Word spread, and now what began as a lunch meeting has turned into a meet-up with approximately 50 Higher Education Drupal users coming together to talk about how they can grow Drupal on their campus and overcome the common challenges they are facing. This is what DrupalCon and the Drupal community are all about!
As Drupal adoption has grown, so has adoption in Higher Education. We recently found that out of 3260 universities that we tracked, over 35% of them are using Drupal including 71 of the top 100 universities like Harvard, Duke, MIT, UPenn, Princeton, UC Berkeley, Stanford, McGill, and many more. That is pretty amazing!
But it makes sense because Drupal provides significant advantages to universities, including support for large scale mulit-site deployments, fit with centralized IT organizations, low end-user training requirements, lower costs, appeal among digital natives and young developers, support for integration with campus authentication and authorization systems, and strong content relation capabilities - particularly taxonomy support for libraries.
I look forward to meeting with this unique group of Drupal users as they learn from each other, and undoubtedly teach us more about the specific needs in higher education. DrupalCon here we come!
At the Web Services and Context Core initiative sprint earlier this month, we attempted to re-scope the initiative so that it was more manageable and less daunting. We decided to spin off one of the major components of what it was trying to tackle into its own separate initiative, the Layout initiative.
The goal of the Layout initiative is to make all elements on the page into contextual blocks that can be rearranged and organized into flexible layouts (and even layouts within layouts) through a drag and drop interface.
Specifically, the initiative breaks down as:
- Contextual blocks: Provide the ability to pass in relevant configuration data to blocks so they can react on something other than the URL of the request.
- Blocks everywhere: Make all page elements—from the site logo to menus to the main page content—into blocks which can be treated the same.
- Multiple page layouts: Choose from either pre-configured layouts such as "3-Column" and "Grid" or make your own custom layouts for pages.
- Partial page rendering: Allow for individual page components to be loaded and rendered independently, allowing for better performance and independent AJAX requests.
- Better UI/UX to manage blocks: A visual, drag and drop interface for creating page layouts and populating with blocks.
This approach to building pages yields a number of benefits, including the ability to customize the look and feel of a site based on a variety of contextual information, the ability to render parts of the page independently for performance, increased consistency, and increased flexibility for site builders to re-use page elements in a number of contexts.
I've asked Kris "EclipseGc" Vanderwater to head up the Layout initiative team. Kris has spent a lot of time working with Panels and Page Manager, and intends to work closely with Earl Miles and other maintainers of the CTools suite. Read Kris's blog post for an overview of the initiatives' goals. Note that like all initiatives, they don't actually materialize unless people decide to help. Please join the discussions in the Blocks and Layouts Everywhere Initiative on groups.drupal.org and help work on Layout issues on drupal.org.
As the Documentation Team lead, Jennifer "jhodgdon" Hodgdon has done a fantastic job of not only keeping Drupal core's API documentation high-quality and consistent, but also of on-boarding new Drupal core contributors through the "Novice" issue queue.
Since documentation improvement patches are always welcome, and since they are unlikely to break other parts of the system, I'm happy to announce the promotion of Jennifer as a Drupal core co-maintainer for version 7 and 8. Her responsibility will be solely around documentation and code style patches, plus occasional help on "emergency" commits such as a required rollback of an accidental patch commit in order to get our automated test suite passing again.
The hope is that delegating responsibility for documentation and code style patches to Jennifer will help increase the velocity of Drupal 8 development. Not only will documentation changes go in faster, it also allows catch, webchick and myself to focus our time on bigger patches.
Welcome to the core committer team, Jennifer! :-)
Last weekend, we held a sprint at the Acquia offices for the Web Services and Context Core (WSCCI) Initiative for Drupal 8. This was an important sprint for the future of Drupal. This blog post provides a high-level overview of what was discussed and agreed upon; the different sprint participants will be laying out more technical details in follow-up blog posts.
Overall, a wide range of experience levels, skill sets, and perspectives were brought to the table, with the goal of the sprint being to clearly define the initiative’s scope, get agreement on what we wanted to accomplish and why, and lay out a clear plan for how to accomplish this.
In attendance were:
- Larry "Crell" Garfield, lead architect of the WSCCI initiative
- Daniel "sun" Kudwien, top Drupal core contributor
- Fabien "fabpot" Potencier, lead developer of the Symfony project
- Kris "EclipseGc" Vanderwater and James "neclimdul" Gilliland, who have helped develop and implement the plugin system as part of WSCCI
- Greg "heyrocker" Dunlap, initiative lead of the Configuration Management Initiative for Drupal 8
- Moshe Weitzman, long-time Drupal core contributor
- Angela "webchick" Byron, community cat herder and Drupal 7 core co-maintainer
- Randy "rfay" Fay, Justin "beejeebus" Randall and Alex "effulgentsia" Bronstein, core developers and sane voices of reason with an outside perspective
- Dries Buytaert (me), project lead
The WSCCI initiative, as envisioned by Larry Garfield, was originally set to address Drupal's web services and flexible page layout capabilities. We discovered that both would require significant changes to Drupal core, and it was difficult to build consensus online, so we decided to get together for 3 days and to flesh out what we actually wanted to accomplish, and how.
At the sprint, we first attempted to articulate all of the problems that WSCCI was trying to solve, which included: multiple page layouts, better UI/UX to manage blocks, partial page rendering (ESI, AHAH), contextual blocks, different response types per delivery callback/URL, plugin system / swappable subsystems, lazy loading bootstrap (convert subsystems to PSR-0 classes), infrastructure for building web services, dependency injection, and so on.
We then did a round of voting where we could each choose 3 of those things in order to try to determine which of those were the most important.
Two things became instantly clear during this exercise:
- The items encompassed under WSCCI really spanned at least 3 separate major areas: Web Services, more robust ESI-based layouts (think Panels only more powerful), and cleaning up our underlying toolset to be a more loosely-coupled framework.
- The underlying architecture to support RESTful calls to Drupal that makes all of the other things possible was deemed the most important thing to focus on.
After a good chunk of discussions, all were in agreement to scale back the scope of the initiative to just the "Web Services" piece, and spin off the Layout/blocks/related-UI parts to a separate effort.
Furthermore, some efforts, such as PSR-0 and Unified Plugin system, were only semi-related to the WSCCI initiative in the first place, and just happened to become relevant for it. Work on those efforts will continue as part of the general Framework community efforts.
Fabien was able to offer a tremendous number of insights as to how various Symfony2 components could help provide underlying structure for Drupal core to support Web Services out of the box. Essentially, most of what the WSCCI team had been trying to work toward, in the abstract, was already implemented within Symfony2. While some implementation details were different than what we had in mind, the end result is almost exactly what we have been trying to accomplish. We therefore agreed that the best way forward was to leverage several Symfony2 components within Drupal as a way to speed progress toward that end.
Some of the concrete benefits that would come out of these changes are
- An underlying framework that is a first-class REST citizen. That means developing web services becomes easier and more efficient, because the plumbing is already in place. An HTML page is a particular case of a REST service.
- Because the core system will be fully REST-ified, we'll be able to improve existing APIs and expose Drupal content in a natively RESTful way. For example, our current AJAX system doesn't comply with REST standards, but with these changes, can be cleaned up to do so.
- That in turn makes Drupal-to-Drupal communication, content staging, content sharing, and a host of other related tasks easier.
- The use of widely used libraries and techniques makes Drupal more approachable to new developers.
Why does this matter?
As it has evolved into an increasingly powerful system, Drupal has gotten increasingly complex and is not as easy to start developing with as it once was. Many developers are nervous about continuing that trend. Managing complexity is a challenge faced by many software projects, and one approach is to use standardized patterns and components.
Due to its long support for PHP 4, as well as some unique needs, Drupal does not take full advantage of standardized patterns and components. The complexity of the custom code that’s used and the non-standard architecture combines to create a barrier to entry for developers new to Drupal (both experienced and novice developers alike).
Meanwhile, the web is constantly changing around us. Web services and mobile are more important than ever, and with that comes the need to have more flexible page and layout capabilities. Now feels like the right time to modernize Drupal’s capabilities to bring it to the forefront of modern PHP systems and web systems in general.
While changing Drupal's core plumbing to address these needs, it's also a good opportunity to do so using more widely understood and modern techniques and libraries. Leveraging the work of a fellow open source community lets us get a jump on these changes to build a more powerful, more flexible, and more easily learnable system in less time than it would take to go it our own.
While these changes may seem large, and some of them are, we believe that they will achieve the right balance between empowering Drupal's design and architecture, and moving toward more modern, standard, well-tested code and techniques to empower a new generation of developers. I hope you are as excited as we are!