Drupal 7 development process retrospective

Drupal 7 was released three weeks ago. Now that we've all had chance to catch our breath, it's a good time to look back and reflect on the Drupal 7 release cycle. Since I'm about to open the Drupal 8 branch, this is an ideal time to inspect and adapt our processes.

In this post, I'd like everyone to chip in and share their thoughts about the Drupal 7 development path. How have things gone? What should we continue to do? What should we change? Should the release cycle be shorter or longer? Should we revisit our policy on backward compatibility? Look back and assess. Think about the interactions and patterns that contributed to both our successes and failures during the Drupal 7 development cycle. Then, taking everything into account, I'll try to set an optimized course for the upcoming Drupal 8 development cycle.

This blog post is only one step in determining that course. It could get a bit unwieldy to discuss this in the comments of a blog post, but that's alright. Let's give it a try and just gather data and insight and try to write a shared story of how things went and what we could improve possibly. If necessary, taking the comments on this blog post as input, I'll organize a more structured follow-up survey to help us prioritize and streamline our thoughts, and ultimately decide what to do. Keep in mind that the decision on what to do won't necessarily be a democratic vote. Good ideas and suggestions count more than mere quantity of affirmations.

One change that is already on its way is the migration from CVS to Git. It will address many of the problems we experienced getting to Drupal 7. I'm quite excited about that and eager to see how Git will change our development process. I'd really like to see us evolve to a Linux kernel development model in which I maintain the official Drupal core tree from which official releases are made, but integrate major changes and bug fixes from different individuals and groups.

I want to be careful not to jump to solutions too quickly as there might be other things that could be changed. At the same time, switching from CVS to Git and adopting how we work together is a humongous change. It might be enough of a change for this release cycle that we don't need to make many more changes. I'd certainly be interested in your thoughts.

In a separate post, I'll organize our traditional 'battle plan discussion' for feature discussion. For now, the goal is just to brainstorm about the development process.

Comments

Berend de Boer (not verified):

Quite happy with the release duration. Big releases shouldn't come to quickly. Given all that is changed in Drupal 7, Drupal 8 should either not come quickly either, or have no API changes, so it could come faster as there is not so much of a learning curve/module upgrade curve.

February 1, 2011 - 22:24
Matt Farina (not verified):

I'm on the opposite side of the fence. I think the next release should be quite a bit faster. Drupal 7 uses xhtml and performs quite slowly on shared hosting. If we wait 3 years to switch to html5 and make some architectural changes for performance it will be a bad thing.

February 1, 2011 - 22:56
Ben Jeavons (not verified):

We cannot stop innovating with each Drupal release. Part of Drupal's success has been its ability to incorporate some of the cutting edge technologies, which call early adopters and keep Drupal from being seen as "stale".
However, we cannot make a release that is too difficult to work with, like inconsistent APIs resulting in bad DX and poor interfaces and interactions that result in an ugly UX.
There should exist a balance between these two ideals, and we should strive for it. How do we get there? Well, where are we going?
Some examples in the innovation camp are HTML5 and streamlined development to production config migration. And by now we have a pretty good idea of what a mess we've made getting where we are. Many people want to finish the initiatives they started.
If we can put together all these ideas we might have a sense of what it'll take to do them all. We can't do them all of course, but we could prioritize and estimate a timeline. And we have quite a bit of available data about our community. Number of issues, time from post, to patch, to commit. These data points *may* help us figure what, as a community, we can accomplish from our goals.

February 1, 2011 - 23:02
Dries:

Great summary on the "state of documentation" -- certainly what I was looking for. Your comment is focused on the problems, but doesn't propose any possible solutions. Any thoughts there?

February 2, 2011 - 03:44
arianek (not verified):

Well, for the 2 "bads"...

The 1st one re: string freeze) I think your questions in response to Jennifer's comments are going to the right place - ie. actually enforce the freezes more strictly or loosen them for everyone. I personally don't care which, but I think it needs to be more one or the other - for the Help text updating, I don't care so much about having to crunch unnecessarily, but the fact that so much more functionality changed after the text was frozen was a problem.

The 2nd one re: getting behind on the online docs);

1. Gap in leadership/team momentum: Perhaps we should create a docs lead handoff process? Of course this could be hard to follow in certain situations ie. if the current lead disappears off the face of the earth.) Having a process for this in place could be useful for important contrib projects too. Momentum relies on leads or very involved docs team members keeping issues updated, etc. so that's just needing more people active.

2. Docs lifecycles: This is something that docs and dev leads need to figure out a plan for and document so everyone knows what needs to be done, and can help.

3. Need to try and finish some of the more high priority docs infra work, and then get a better delegation workflow as well as willing docs team members) to catch up on the massive maintenance debt for the online docs. So infra + recruiting and motivating more docs team members especially people willing to take more responsibility on and help coordinate docs sub-areas).

February 2, 2011 - 04:24
catch (not verified):

Upgrade documentation, I think we have a real problem with the commit -> 'needs work' + needs documentation pattern. People don't want to write docs for patches that don't get in, but it's very easy to forget once the patch is committed, sometimes there's followup work to do on a patch other than docuementation - like fix regressions or backport to previous versions etc.

If we coul
d move the individual API change documentation to individual nodes instead of one huge page, we could then create stub nodes either when patches are RTBC or just after commit, and those individual pages can be marked as in
complete/outdated like other handbook pages. A multi-value node reference field to the original issues) would be nice too ;)

February 2, 2011 - 04:51
arianek (not verified):

I definitely think that if there's a way to get documentation to be more integrated in the workflow (rather than an afterthought), that'd be a great start. I feel like this is almost asking for a bit of cultural change within the development community, which is something that might take some more significant planning/discussion/publicity to do in a successful way. It would be unfortunate to try and make this sort of a change, and not have it adopted.

I also think infra-wise, it seems like there are things that the online docs and api docs could benefit having; the online docs would benefit wildly from proper versioning (Jennifer's got an issue started here to support this http://drupal.org/node/995362) and I could see the rationale behind having some tagging and a better workflow for tracking necessary API doc updates as being useful.

February 4, 2011 - 04:48
Larry Garfield (not verified):

Coordination.

Drupal 7 was in some ways the most coordinated Drupal ever, with Angie acting as the most proactive and out-there comaintainer we've ever had. In other ways, it was the least coordinated as there were things going on that I as a subsystem maintainer knew nothing of... sometimes even in the DB system itself! We've gotten too big for a two level structure Branch Maintainers and Everyone Else).

That top-level coordination needs to be more of a team that is on the same page rather than one super-woman trying to herd all the cats at the same time.

As a subsystem maintainer, it still irks me that there is no clear definition of what I do or what responsibility/authority I have as such. Angie has given me a verbal answer that boils down to having a bat-phone to her for DB-related issues, but I would still rather have a clearer picture of what our structure is because we clearly have one; we just try to avoid talking about it). That would allow people to "own" a sub-project. There were things we wanted to do for the DB layer that didn't get done because I never knew if I had the authority to just make a decision, and didn't get any feedback from higher up Dries and Angie) despite repeated requests. That sort of situation should not be allowed to happen.

Being able to fully leverage Git esp. with the Phase 3 plans) would make that tons easier.

You've mentioned before that we'll probably have 2 D8 maintainers, a "framework maintainer" and a "product maintainer". I am very +1 on that. But what is the line between "reviewing and prioritizing patches" and "reviewing and prioritizing concepts"? Those are very different skills, and someone who is good at one may not be any good at another. But it's very important to know who to go to for each one when you have a question.

We don't want to have people waste time on something that may or may not be accepted; that's called spec work, and it's just as sucky for developers as it is for designers. :-)

What worked well is that Angie was almost omni-present and was overall extremely responsive to discussion and brainstorming from key people. What didn't work is that there was only one Angie, and she had to eat sometime. :-)

We also need to be very, very explicit about our architectural priorities for D8. I have more to say on that, so will make it a blog post of my own shortly.

As far as timeline, I'm in favor of targeting December 12, 2012 for D8's release. That's a little under 2 years, and just in time for the world to end. :-) It gives us enough time for the many plans we have for D8, but it's not as long as the drawn-out D7 cycle.

February 1, 2011 - 23:26
Dries:

Great feedback.

As a subsystem maintainer, it still irks me that there is no clear definition of what I do or what responsibility/authority I have as such. Angie has given me a verbal answer that boils down to having a bat-phone to her for DB-related issues, but I would still rather have a clearer picture of what our structure is because we clearly have one; we just try to avoid talking about it).

It's impossible to have one clear definition that is applicable to all sub-system maintainers -- it is also very uncommon to have one set of rules. For example, what decisions are made by a company's CEO versus his direct reports? Or by the lead architect versus the senior engineers? It takes constant communication, and implicit understanding based on past experience and level of trust.

In other words, the decision making policies are different for every person and for every project. It is varies on a case per case basis, and therefore can't be captured in one clear definition.

Where things might have broken down is with the lack of communication with either Angie or myself, not with the lack of a set of documented decision making policies. I think sub-system maintainers can have quite a bit of decision-making authority as long I feel we're completed aligned.

In this specific case i.e. Drupal 7's database abstraction layer), we probably should have had a monthly meeting to discuss progress, to get alignment, and to make decisions. That is certainly something that I can get better at in the Drupal 8 development cycle. In addition to that, you should continue to have the constant conversation in the issue queues.

February 2, 2011 - 02:49
Larry Garfield (not verified):

"Different for every project" simply doesn't work when every issue is, realistically, it's own mini-project. That means that I don't know issue-by-issue whether I can realistically make a decision and have it "stick' or if you'll come back with a "eh, think about this more" comment (which always results in the issue taking another 4 months before you even look at it again, in part because by that point the people IN the issue have already come to a conclusion). That lack of clarity is extraordinarily frustrating and slows down development.

And it's not just me not knowing. It's everyone else I work with not knowing what my position is. And, especially, what level of trust you as project lead have in subsystem maintainers in general. I genuinely cannot answer that question based on past experience, either in the DB queue or elsewhere, and that is a problem.

Definitely active, continuous communication is important, and in D7 we had far too little of it. We should have more of it going forward. But at the same time, there were too many balls in the air for you and Angie to be omni-present and "grok" each issue (even though Angie made a valiant effort).

But without a clear, explicit mandate the implication was that nothing went in unless one or the other or both of you grokked every issue, because there was no clear, publicly stated trust placed in subsystem maintainers.

I as a subsystem maintainer don't know where I stand, and by implication no one else knows where I stand either. That caused serious, concrete problems in the queue that held back several patches.

I also have to challenge your "well companies don't have a clear structure either" statement. That's simply untrue. Companies that have a poor structure or too heavy of a process tend to fail, yes. But companies that have a good, clear structure and effective process succeed. I've seen that first hand. "Just do whatever and we figure it out" is a sure-fire way to lose people, clients, and hair. There is a very clear distinction between what a "lead architect", "senior engineer", and "project manager" do. There may be overlap, but in an effective team everyone knows what is expected of them, and what to expect from other people.

Right now, we don't have that.

February 3, 2011 - 18:56
Thomas Svenson (not verified):

As I see it, a sub-system maintainer should "own" it and work as a filter for the main maintainer.

I disagree with Dries that it is very different for each every person/project. Sure there are differences, but there are also a whole bunch of basic "rules" that can be applied in every case.

I think that for D8 it will be important to find a common formula and guideline for the role definition for sub-system maintainers, including reporting, documentation and the authority they will have over their projects. That also includes decisions needed to be made.

With a proper role definition it will be easy for everyone to know how things work.

The goal should be to have as few unique rules as possible.

February 7, 2011 - 08:02
PWolanin (not verified):

TO build on what Larry said here about each issue being its own little project, it would be nice to step back a bit at the beginning of bigger or more controversial changes and make sure the higher level discussion about architecture happens before coding starts in full.

Then, once an issue has some momentum, think about clearly and specifically delegating a lot of final decision-making power for that one issue to one or a very few people. That might help avoid issues getting bogged down or derailed when they are 90% complete.

February 10, 2011 - 03:26
Dries:

As far as timeline, I'm in favor of targeting December 12, 2012 for D8's release. That's a little under 2 years, and just in time for the world to end.

I kinda like that. Let's discuss this some more to see if it makes sense too. :)

February 2, 2011 - 02:52
Marco Sousa (not verified):

I like that date too. 12/12/12

February 2, 2011 - 20:14
Matto (not verified):

I think 12/12/12 is an excellent launch date for D8, from a marketing & promotional perspective.

(Of course, it would be better if we skipped D8, 9, 10 and 11 and named the next release D12, but Eminem might sue for trademark infringement ;) Actually, there's a PR opportunity right there: "Eminem raps web wizard's knuckles over name conflict".)

A few quick thoughts:

* We could turn Doomsday into DrupalDay by promoting the D8 launch on 12-12-12 with the theme: "The web world as you know it is about to end."

* The 12-12-12 date and it's equivalents will become (and already are) very popular search terms - we could make a big SEO effort to create a page or site about the D8 launch that tops the Google results and generates traffic and awareness. (The story behind that effort would be another PR opportunity.)

* The 12 days of Christmas is a fairly well known concept, so you could plan "the 12 days of Drupal", starting on Dec 1st and finishing on the launch day 12-12-12. Each day would feature some event or PR stuff about a different feature/benefit of Drupal.

* There's never going to be a 13-13-13. We might as well piggy-back on the last cool date of the century.

But... if you promised/publicised that date, you would really have to launch on that date. No ifs or buts. No slush, only freeze.

And two years seems a reasonable development period, from my relatively ignorant perspective.

February 4, 2011 - 04:30
Larry Garfield (not verified):

And of course I goofed. It's 21 December 2012, not 12 December. Still, "end of 2012" is a good target date, IMO.

February 4, 2011 - 06:02
Matto (not verified):

Alternatively, you could adopt a two-year (or one-year or three-year) cycle for major releases and make December (D-cember) the common launch month for Dx, as follows:

"D-cember D8" 2012 (Drupal8 launches on December 8th 2012)

"D-cember D9" 2014 (Drupal9 launches on December 9th 2014)

"D-cember D10" 2016 (Drupal10 launches on December 10th 2016)

That would give Drupal some "ownership" of December (at least in Nerdistan) and give all Drupal stakeholders a handy mnemonic for remembering or working out when future major releases of Drupal will ship.

This convention could work for 23 years and 11 major releases.

AND, if we really want to piggyback on 12-12-12 (see my other comment) we could release D8 on Dec 12th 2012 and then start the system outlined in this comment with D9 launching on Dec 9th 2014.

This pattern & mnemonic would have many benefits for the promotion of Drupal.

I dare say it would also remove one burdensome task from the Drupal devel cycle, namely the discussion about, and adoption of, the next release date.

February 4, 2011 - 05:26
Dries:

Very geeky, but kinda cool! :-)

February 4, 2011 - 19:02
Matto (not verified):

Correction to my maths: The D-cember Dx launch date format with a 2-year release cycle could last for 46 years until Drupal31 is released on D-cember D31, 2058.

February 4, 2011 - 22:41
adrinux (not verified):

Whilst 2 years has the right feel too it, I wonder about the wisdom of trying to get releases finished off and done right on top of the 'holiday season'. There's never a good time, but December doesn't seem like the best choice.

February 8, 2011 - 15:35
Matto (not verified):

Here's another launch date format that would "work" for the next five major Drupal releases (and only one would be near the holiday season):

* 8-8-2013 (Thurs) D8 launches 2 years 7 months after D7
* 9-9-2015 (Wed) D9 launches 2 years 1 month after D8
* 10-10-2017 (Tues) D10 launches 2 years 1 month after D9
* 11-11-2019 (Mon) D11 launches 2 years 1 month after D10
* 12-12-2021 (Sun) D12 launches 2 years 1 month after D11

These double-release-number launch dates would be very promotable and sloganable, e.g.

"Can't wait for 8-8"
"Get in line on 9-9"
"Launching again on 10-10"
"11-11 is Drupal heaven"
"Nothing rhymes with 12-12 but that's when D12 is due"

When D8 launches you may decide the proposed 2 year interval to the D9 launch should be changed to 3 years 1 month, or 1 year 1 month, and that would still work.

For D13 you would set a release date on the 13th of a suitable month, e.g. Wednesday 13 March 2024: 13-3-2024 (or 3-13-2024 in the US) ... and so on, until you run out of days-of-the-month after Drupal 31. But by then, the Great Pumpkin will have returned and we'll have a decimal calendar.

By the way, Drupal7 could have followed this day-of-the-month approach by waiting 2 more days to launch on Friday January 7, and thus would have created a much better backstory for future Drupal promotional guff.

Pre-emptive note to Captain Obvious (I know you're out there, waiting to pounce):
Yeah mate, I know that marketing considerations shouldn't dictate the release cycle. So save yourself a post.

February 9, 2011 - 05:33
Nigel (not verified):

I think at sometime in the future backward compatability will have to be considered if Drupal continues to grow.

As an example, if you assume 1 million live Drupal sites at some point in the future I have no idea what the actual number is now).

Each site takes say 20 hours on average to upgrade a wild guess).

That's 20 million man hours of upgrade work.

Assuming a developer has 27 effective work hours per week and they work for 46 weeks per year.

20,000,000 / 27 / 5 /46 = 3,220 man years

My point is at some point in the future, with the growth of Drupal, the whole development community will be consumed by upgrades to existing sites. Or have I got something wrong?

February 2, 2011 - 01:10
Ryan (not verified):

I'm not so sure we can make those assumptions... specifically that every site is even going to bother with an actual update. I'd be interested in seeing statistics on the number of sites who consider updating from one version to the next critical / a priority / interesting / unnecessary. I know every site I've "updated" from one version to the next has been a rebuild to accommodate the changes in core architecture and contribs from one version of Drupal to the next. If this really became an issue, I'd think it would be more realistic to treat major versions of Drupal as separate applications with content migration scripts from one to the next. In fact... it may be the Migrate module already has a solution for this. : ?

Additionally, does backwards compatibility mean a Drupal 8 site running on a Drupal 7 database? A Drupal 8 module working in a Drupal 7 site? A Drupal 7 module working in a Drupal 8 site? All of the above with mixed database versions?

I've never really seen that defined... but it all smells like a waste of time to me.

February 2, 2011 - 05:28
catch (not verified):

Every upgrade I've done has been more or less a ground up rewrite of the site as well.

I posted a Drupal 8 issue (which I can't find now) to discuss dumping hook_update_N() for major versions, and using migrate instead. This would give us the ability to 'update' from live sites, incrementally, with rollback, all the other lovely things that migrate module does. Alpha and beta releases of core would still need hook_update_N().

This would also be a solution for skipping versions - build the Drupal 7 code + config, migrate the Drupal 5 content over to it.

I think this would be a much better workflow for large sites, the challenge would be making it work great for non-coders with simple sites, since on sites where you really can just run update.php and come out the other end with a functioning site, it'd be overkill. But I should find that issue and link it, since this IMO this is more about Drupal 8 battle plans than Drupal 7 post-mortem.

February 2, 2011 - 17:50
adrinux (not verified):

Migrate/import sounds like a killer feature! It might also open up the possibility of allowing out-of-the-box 'upgrades' from other cms systems.

February 8, 2011 - 15:56
jhodgdon (not verified):

One complaint I had about the development process for D7 was the freeze/slush process. It seemed like the code freezes of various types APIs, UI, strings) weren't used/enforced consistently.

For instance, there was a string freeze announced very early on about a year before the actual release happened I think, or even more?), and the doc team worked very hard to do a major edit of the core module in-line help pages to meet that string freeze date. But then that freeze seemed to be forgotten, and the real string freeze was not until the beta release. APIs were also changed way past the original API freeze date probably mostly for good reason)... but it lent a feeling of arbitrariness to the whole procedure, for me -- it felt like "strings are frozen unless we decide they aren't", and "APIs are frozen unless we change our mind".

I am also not sure if the freeze dates for API, UI, and strings made sense -- the originally announced freeze dates were very close together. It's difficult for a UI to coalesce until the API underneath is stable, and it's difficult for usability/doc/accessibility people to review the UI and on-screen text until it's been built so they can see it, and it's difficult to write in-line help until the UI is settled. So the original proposal of freezes was probably too tight, and didn't leave enough time for review, as developers saw the freeze dates as the last moment to submit something.

I guess what I'm trying to say is that the freezes either should be more consistently enforced and set up more sensibly, or they should not exist at all.

On a more positive note, one thing that I think was a HUGE win in D7 development was insisting on tests for most commits. Because of that, I think D7 and future versions will prove to be much more stable and less buggy than D6 and prior versions were on their initial release dates. Let's continue that -- yes, it's a bit of a burden on developers, but my experience with testing my contrib modules as well as writing some core tests) has shown that in the process of writing tests, you find so many bugs you didn't know were in your code, it's definitely worth it. Plus it helps to prevent regressions and unexpected side effects. A big win!

Another thing I think was really good in the D7 process was that there were usability and accessibility people heavily involved in D7 development, and their perspectives were listened to and acted upon at least, that is what it seemed like to me).

Well, Dries -- you asked for rambling -- I think I've fulfilled that objective at least. :) Let's make D8 an even better process and an even better Drupal!

February 2, 2011 - 01:42
Dries:

I agree that we've been bad at enforcing code freezes -- including the string and UI freezes. We have always been bad at it, frankly. It is a repeating pattern, and therefore very worthy of discussion.

We certainly could have been more strict about code freezes, but it would have negatively impacted the project in other ways. We always end up with hundreds of difficult micro) trade-offs to make. Do we allow this in and fix a major WTF, or do we draw a hard line and say 'no'?

So let me ask two fundamental questions for all of us to answer:

  • Do you believe we made the wrong trade-off more often than not?
  • Is the notion of a strict code freeze after a long development cycle unrealistic and a recipe for frustration?

These are difficult but important questions.

A code freeze has the interesting side effect that it helps kick us in the right gear, as demonstrated by the Drupal 7 commit history graphs ...)

February 2, 2011 - 04:04
Tom Geller (not verified):

I think you have to say "no" more often and more firmly than occurred with Drupal 7. For example, the issue I linked to in my other comment was just a prettification: Leaving things as they were wouldn't have broken anything.

I believe that firmer code freezes will *increase* the good side effect that you pointed out, while promoting trust among those who depend on the schedule.

February 2, 2011 - 18:39
catch (not verified):

Said this below, but short API/feature freezes to keep the critical issues count under control, then a regular-ish unstable/alpha/beta cycle (all released with 0 known criticals) would mean that when we do have a full freeze, there would be much less chance of getting tied up for months with accumulated critical issues. I think we'd still see the acceleration as things get close to alpha, that's about scheduling/urgency rather than how frozen things actually are.

It's the sinking knowledge that you might have to wait a year before it's even worth looking at that nice patch you were working on again that creates so much pressure from people to squeeze things in after freeze.

With D7 we were unable to release an RC for months and months, despite being in 'freeze', due to upgrade path and other critical bugs. While we didn't have an RC, we were still beta, so things would keep slipping in, and adding to the critical bugs count (*cough* filter module).

If we can make the final freeze short and sharp - pretty much only at RC stage, which was the intention of automated testing but didn't really work out, then the decision to release beta/RC would be around comfort in features/API, and wouldn't necessarily mean "we're about to start a hard slog where you won't be getting any fun patches in for the next year and three months".

This is going to mean more discipline during thaw in terms of tidying up after ourselves as we go along, but I don't think anyone wants another 16 month winter.

February 3, 2011 - 06:53
cweagans (not verified):

I think it would make more sense to say that code freeze is when alpha1 is released.

In D7, a lot of commits did require tests.

In D8, I'd like to see test coverage increase. I'd also like to see an aggressive documentation policy - basically, if I were to write a huge new system for Drupal, I would be required to write documentation for it, both from an API standpoint and from an end-user standpoint. It makes sense to do this because nobody is going to know a system like the person that designed it. I wrote a couple of blog posts about it: http://cweagans.net/blog/2011/1/31/building-better-documentation and http://cweagans.net/blog/2011/2/2/building-better-documentation-part-2

I'd also like to see "core" docs and "contrib" docs - core docs would be a nicely packaged set of documentation that gets shipped with Drupal. The documentation for CodeIgniter is what I'm kind of envisioning for Drupal.

February 3, 2011 - 11:24
arianek (not verified):

re: http://cweagans.net/blog/2011/1/31/building-better-documentation and http://cweagans.net/blog/2011/2/2/building-better-documentation-part-2

Like rfay has noted, we've been discussing this since December - I pitched a core conversation on it which wasn't accepted, but you can see the notes in the gdoc on this comment http://groups.drupal.org/node/118474#comment-391894

There are other things that need to be taken care of before I can personally put more time into this initiative (which needs a lot of discussion, as I think it's got both some merits and some drawbacks), but would like to address it sometime this year.

If you're interested in helping with these initiatives, please do follow http://groups.drupal.org/documentation-team discussions and the issues Randy posted, there's a small but committed core of docs team folks who are trying to improve matters!

February 4, 2011 - 05:09
Jeff Geerling (not verified):

I think one thing that could help a bit more with the D8 cycle is a more formalized and specific sprint structure.

Rather than a 'core sprint,' we might be better off with a 'theme system sprint' here, a 'ux sprint' there, etc.

Having regular sprints and multiple sprints) would be awesome, if at all possible. Once a month, maybe, on two days of the week, or in some format that would allow the maximum number of contributors be able to participate online, a core maintainer could work with everyone on a particular subsystem.

I know that it was often dizzying at times to try to keep track of five or six areas I was interested in, simply because things were being committed in a haphazard fashion. If there could be a little more focus on specific areas at specific times, that would maybe bring a bit more sanity in development.

My 2¢.

February 2, 2011 - 02:11
Tom Geller (not verified):

I can't stress how important @jhodgdon's statement is.

"One complaint I had about the development process for D7 was the freeze/slush process. It seemed like the code freezes of various types APIs, UI, strings) weren't used/enforced consistently."

I'm pleased with all other aspects of the development process, but the "UI freeze" was an absurd and damaging joke. Look at this discussion for one example.

This time around, violation of the code freezes had damaging effects for only a few people -- I was one of the unlucky ones. With Drupal's growth, next time they'll affect far more people.

These are not trivial matters: If a company has to change its plans because core maintainers don't enforce deadlines, workers will be laid off. Families will go hungry.

As the saying goes, "The higher the monkey climbs, the more you see of its bum."

We're moving into a larger world and therefore have to take our commitments more seriously. Our actions have greater effect.

February 2, 2011 - 03:05
Dries:
February 2, 2011 - 04:07
Mukhsim (not verified):

I suggest accepting the Canonical release schedule type. Perfectly fits product/framework releases. Release product updates every 6-12 months on stable framework API and release major API updates every 2-3 years. Security patches would apply for the current and previous major release giving organizations about 5 years of supported lifetime for the product. UI/HTML5 crowd will be happy with quick releases that allow to polish up the product.

Mukhsim.

February 2, 2011 - 03:17
momendo (not verified):

I too like the Canonical model. Their release cycle is so predictable, you can bet money on it. I believe it sets the right expectations and what is achievable in that timeline. Big grandiose ideas get broken down into chunks. Slow movements become mobilized. The question one may ask, "when will it be done?" -- well, we know the answer to that. No ambiguity. With a set time, we can schedule everything else around it predictably.

February 2, 2011 - 04:28
catch (not verified):

You can't have separate framework and product release cycles, unless you have separation between framework and product, Drupal doesn't.

We have complex interdependencies which will need to be cleared up, that will take a release or two. common.inc, system module etc.

We also have large areas of core that are neither framework nor application, but firmly 'framlication' - field API and UI, node module etc.

Not to mention that UIs are really part of the API - if my module form_alters a core module's form, and the structure of that form changes dramatically, then that's an API change for my module. It might be a smaller change than a new database layer, but there's very little in core that can be touched without having /some/ effect on contrib somewhere.

So it would be good to get to a point where something like this is feasible (whether we wanted to do it or not, decoupling many things would be good), but the ground work has to be done first.

February 2, 2011 - 18:42
Mukhsim (not verified):

Already the best modules are the ones that have successfully decoupled API from UI (views, for example). The worst one is probably Ubercart and that's the precise reason for a complete rewrite despite a huge demand for the existing solution.

There are only 7 core modules required for D7 to function. Everything else can be moved from /modules to /profiles/[profile_name]/modules/.

It takes time for APIs to mature, and it is best if they do it outside of core. Distros allow for this to happen without putting extra burden on the core maintainers. Having profile maintainers would probably help, because they can improve the product without messing up the architecture.

Distros can be released every 6 month and over time when they provide mature enough API which is universally demanded (Services, for example), only then they can be moved to /modules. Even then it is desirable to have related UI in /profiles because lots of stuff in enterprise is done with drush.

Smallcore+distros can provide the solution so demanded both by enterprise in tems of stability and support and startup/site builders who want latest and greatest without having to wait for it for 2-3 years.

Demand for this is out there and people will be willing to contribute if they understand the benefits of the changes and can see the results of their work quick. Expect 10 times more developers joining after git migration.

Mukhsim.

February 3, 2011 - 02:01
Paul Kishimoto (not verified):

I don't agree with Mukhsim's idea of API releases on one schedule and "other" (?) releases on a faster schedule; but the idea of timed releases is definitely worth some study. I agree Drupal is different in many ways from a Linux distribution, but that doesn't mean there can't be a unique, Drupal-specific way of doing timed releases.

In watching the D7 development process I was first excited by the D7CX and then disappointed when the commitments went largely unfulfilled. I don't think that's due to any failing of the core developers; but perhaps the uncertain length of the development process made it difficult for module maintainers to know when to update their code. To the extent that users of Drupal are also users of contrib modules, that motivates adjusting the process.

If, as momendo suggests, bigger changes were broken down into chunks that fit a (frequent) regular cycle, contrib would have a much easier time keeping up. This seems possible; none of the major D6→7 initiatives can be characterized as atomic (indivisible).

One feature that helps Ubuntu sustain regular releases is strong dependency management in the apt packaging system. If a user has a package that depends on "python <= 2.6", then a new Python 2.7 won't be installed until a new version of that package is released…even if the only change in the new version is an updated dependency on "python <= 2.7". The user is always assured of running the latest, working combination of packages from "main" (~core) and "universe" (~contrib).

If each Drupal version had a smaller set of API changes, then updating a module for a new release would, in many cases, only mean bumping a dependency.

Again, Canonical's way of doing timed releases will not work if naïvely applied to Drupal, but timed releases are desirable.

February 7, 2011 - 07:35
catch (not verified):

Here's mine:

On release cycle length, I like Larry's "just under two years" - that's enough time to let things bake in, but short enough to maintain momentum. Drupal 7 was really a two year release cycle, plus an extra year tacked onto the end.

For me the main reason things dragged at the end, was we let critical issues get completely out of control:

- We opened 200-300+ critical issues for tests and API deficiencies that had the major priority existed should have been 'major'.

- Even with those, there were about 150 /real/ release blocking issues around code slush period that had to be worked through one by one. The fact we had a critical queue of 200+ meant the 30-40 really gnarly issues got lost in there until the very end.

- the more things slipped, the more people realised that X change was going to have to wait another 2-3 years to get into Drupal 8, so a lot of us stretched the freezes as far as we could, in many cases this led to even more critical issues.

To fix this:

- For Drupal 8 we're starting out with the major priority so there's no need to abuse critical for "really ugly" or "really important" any more.

- I'd also like to see is try regular unstable/alpha/beta releases throughout the cycle - ideally keeping the critical issues count in single figures, or possibly even 0 when these come out. For example we could have mini-freezes of a week or two where we get the critical queue back down to 0 again, and non bug-fix patches get pushed back to the next unstable release. This is something we need to agree on in advance so everyone has the right expectations.

This way while we might still have major ongoing work on, we'd hopefully not build up such a huge, unmanageable, painful backlog as we did in Drupal 7. This would also give a better target for both contrib maintainers, and potentially very brave site builders (of which we had some for Drupal 7) chasing HEAD.

I'm not sure about the strict sprint model others have mentioned, although areas of focus at certain times alongside the daily patch scramble might not hurt.

On Larry's points, while I think there is a danger to adding too much structure to MAINTAINERS.txt roles and responsibilities - people go AWOL after all, there's definitely a need for (more) core maintainer feedback on large patches before they reach the RTBC stage. The RTBC queue (and to an extent the reviews queue, it's not just about commits) operates on a LIFO basis most of the time - the longer an issue has been RTBC, the longer it will stay RTBC - while occasionally issues go from first post to commit while at least one core developer is sleeping.

It's frustrating as a patch author when your patch is RTBC for months except for re-rolls and doesn't get feedback. It's also frustrating when you wake up, see a commit to an area you're interested in, and realise the issue was posted after you went to bed the previous night, especially if you spot an issue in the patch and have to re-open it for a rollback or re-working.

On backwards compatibility, that's a joke right? But, I think we should think seriously about /forwards compatibility/ - i.e. try to backport new APIs from Drupal 8 if they can be done so without breaking existing APIs or stability in Drupal 7. Would have to be case by case, sometimes core, sometimes contrib. http://drupal.org/project/dbtng is a great example of this for D6. Actual core backports from Drupal 7 to 6 have been very limited at least during the latter half of the release cycle, but it feels like automated testing should help at least somewhat to make this a smoother process.

Also the current process is broken - http://drupal.org/node/1017672 is one of our most serious critical issues, and mainly exists because we collectively forgot to backport something to Drupal 6, whoops!

February 2, 2011 - 04:25
catch (not verified):

A couple more things it'd be good to put in place early:

- At one point during the Drupal 7 release, we had automated re-testing of patches, this ensured that stale patches were marked stale by the test bot before humans tried to apply them, but it's no longer running. It would be good to start the Drupal 8 cycle with this back in place again.

- During the Drupal 7 release cycle, I came across countless performance regressions - either compared to Drupal 6 due to structural changes, or just day to day regressions. Nearly all of these were found after patches had already been committed, sometimes they were nearly irreversible, at least not without completely removing new features.

At one point I was profiling core at least once a week, and rather than working on improvements, barely keeping up with the regressions that were going in.

This is an unsustainable level of effort, and I think to fix it we should start making some kind of performance gauntlet a condition for commit - currently that gauntlet only has to be run for 'performance' patches - whereas tiny little feature or cleanup patches are the ones that can easily do the damage. In other words I'd like to see a shift of benchmarks away from justifying that a particular patch is a performance improvement, to benchmarks and/or profiling to justify that a patch isn't a performance regression.

A full benchmarking suite and automated testing would be great to evaluate overall system health, but it it's not necessarily useful for individual patches. One thing we could try would be posting before/after xhprof output showing cpu and memory usage for changes to functions in the critical bootstrap, page rendering, or similar paths. In the same way as we developed an unofficial standard of posting tests that fail without a patch, then a combined path tests that passes.

February 2, 2011 - 04:45
Larry Garfield (not verified):

I disagree that the performance gauntlet only had to be run for "performance patches". There were other patches where performance concerns were raised, sometimes justifiably sometimes not. However, the underlying point I agree with completely: We treat performance as an after-thought, sometimes, in some patches, rather than as an integral part of our process.

That needs to change.

Whether that's an automated benchmarking suite or something else I don't know, but we need to make performance improvement a key part of the Drupal 8 cycle at every level. This is both a process shift and a cultural shift. We even need to be able to say "sorry, this feature is cool but just too expensive on performance"... maybe even for features we have in core now that have to be removed or significantly redesigned on performance grounds.

We also need to develop a better understanding of the performance dynamics of the language we're working with. Where PHP itself is slow, where PHP itself is memory-intensive, etc. That needs to inform our architectural decisions.

We also need to collectively accept that performance and scalability are not the same thing. Drupal 7 is without a doubt the most scalable Drupal release ever... if you're scaling up. Witness Acquia Gardens, Examiner, House.gov, etc. That's awesome. But how easily you can swap in memcached and MongoDB and Varnish means diddly squat for 95% of the market that is on a single server with a stock MySQL and no opcode cache. Performance is making Drupal faster for *them*, to make the 100 visits a day sites lickity fast, while still allowing the big boys to swap in all of the high-end tools to make Drupal scream.

February 4, 2011 - 06:18
catch (not verified):

I agree that the main issue is performance is usually an afterthought and not baked into the process. I also agree the community is collectively better on scaling than performance. More or less this comes down to more developers having spent time dealing with a site falling over, than dealing with just slow response times or high resource usage (and the respective attention given to these by site-owners being widely different too).

On Gardens I seem to remember we've had this conversation before - it's extremely close to shared hosting, much more so than pretty much any other Drupal installation except the ones that actually are on normal shared hosting.

Gardens is running a very large number of sites, on x amount of hardware - many, many small databases instead of one huge one etc. If you have 30,000 sites serving 1m requests, then that's 30,000 variable, theme, schema and other caches to build, unlike one site serving 1m requests. There's also a massive difference between serving 1m requests across 3,000 pages, and 1m requests across 300,000 pages.

I'd like to see the same attention given to identifying certain archetypal performance profiles as there has been to trying to identify audience profiles (with some of the same caveats, but it would help to frame these conversations better).

On shared/single server hosting, Drupal can scale very well with normal page caching or boost if you have a certain kind of site - where things get difficult is when you have a fairly complex site with module bloat and/or lots of authenticated traffic and/or lots of content. Most of those issues are affecting sites across shared hosting, small VPS, a single dedicated server, or on 2-3 servers too.

In terms of priority, for performance I would put things in this order:

- 1. improvements for everyone.
- 2. improvements for premium shared/small VPS/small dedicated.
- 3. improvements for large/massive sites.
- 4. improvements for shared hosting.

Because there are plenty of #1 if you look carefully, #2 has a decent chance of helping things for #3 or #4 (if not both), and when it comes to it, #4 has the option to upgrade to #2 if they really have requirements that go beyond shared hosting.

If we put in changes into core (especially low level changes that aren't easily swapped out or require expert knowledge to do so) that improve things for shared hosting but are a loss on anything else, then rather than the $5-$15 jump, people will be making the $15-$100/month jump much faster - and it's that jump which was one of the main things that got me looking at Drupal performance issues in the first place - for a site that has zero budget except for donations but a very busy community.

Overall what I want to see though is much more emphasis on actual data, and throughout the process of working on changes rather than at the last minute or after the fact - then at least we'll know what trade-offs we're actually making more often, and like you say we need to start seriously discussing whether certain features are worth adding or keeping if they come along with performance regressions.

One more point of disagreement, I'm not sure if this was just wording or not, but the implication in your post is that performance isn't really an issue on large scale installs, I can't agree with that. If you are running a large site and consistently need to serve uncached page requests in < 2 seconds then memcached, varnish and the rest only gets you a very small way towards that, and since there are hard limits in terms of processor speed, this is why we're seeing people looking at multi-threading, HipHop etc.

I've seen high-end Drupal 6 sites on 4+ servers that are using 60-80mb of memory in PHP for each request with an op-code cache, and with a good second or two spent in page execution time. Some of the culprits listed here http://bit.ly/hscCMg - a lot of those issues sadly opened in the last month or so, wish I'd found them a year ago.

Also a lot of issues with slow response times are around cold starts, the standard response to an issue that only affects a cold start is "it doesn't matter because it's cached anyway" - it's those things like menu rebuilds, theme registry rebuilds and the rest that can take down both sites on shared hosting and potentially the large sites too, so we need to revisit that position.

This is going off-topic a bit but it's an important discussion to have, I'll try to start a conversation on it elsewhere.

February 5, 2011 - 15:56
sunward (not verified):

There should be more time between releases. This would allow a better product and time for modules and add-ons to catch up in previous editions.

Currently, version 6 is stable, but it is way too soon to upgrade to 7. Many modules have not been updated to work with 7 and Commerce is still in alpha. I can't even look to upgrade my ecommerce site until later in the year or even next year.

Get 7 stable and working with all the modules with it.

February 2, 2011 - 07:48
peach - sooperthemes (not verified):

I think there should be more attention for the contrib update process. This time D7 upgrades in contrib are much further than around the D6 release but still I'm not upgrading my site because of several important modules.

Maybe there should be a sidebar widget or frontpage block that highlights contrib projects that need testing and reviewing etc.

February 2, 2011 - 10:41
swentel (not verified):

I can agree with Jeff Geerling's comment. I even wrote a blog post about something alike (see http://ow.ly/3MC1U). To know which sprints we want to organise or to participate in we should have a roadmap of things we actually want todo. As an example, take a look at http://fedoraproject.org/wiki/Releases/15/FeatureList - which gives me a clear overview from what I can expect for the next release in my linux distro.

Once we agree on what we want to add or fix in Drupal 8 and we create a roadmap, all people interested to help know exactly which areas to tackle. It also helps to jump in after a couple of months if you have been out for several reasons. During the first year and a half of the D7 cycle, I had a pretty good look on the issue queue, because I followed a lot, wrote patches, discussed on IRC etc. Because it got busy at work, I more or less took a brake, came back after half a year and had to work my way in again, because a lot had changed, but I had no real hold of why x got in, or changed etc, what to work on next etc.

A roadmap would also benefit people looking from a further distance: companies who use Drupal everyday to make money from, clients who are interested in maybe using it etc. Following progress would be bit easier and they can anticipate on them or not depending on what we're (not) doing.

Finally, on a much asked question 'When is Drupal x ready', instead of answering 'When it's ready' or 'When all criticals are 0', we can lead them to like say drupal.org/roadmap-8 instead.

February 2, 2011 - 10:48
Scott (not verified):

After almost two years of being immersed in using Drupal (mainly 6.x, recently started learning 7.x), noting that I'm coming from a decade's worth of both design and development experience, I have a confession fit for this thread.

I almost abandoned Drupal for Django, on the basis that the latter is a true framework and not a content management framework.

I really love the idea of having a "blank canvas" to work with, quickly pulling in stable modules to expand functionality as needed.

Because of the heavy work I would need to do to add stable features that I take for granted in Drupal into my Django-based project, I stayed with Drupal.

I humbly suggest that the Drupal community use the period between major releases to really think about Drupal's line between a true framework and a content management system.

The combination of 'small core' (or 'Drupal kernel') with optional installation profiles (perhaps including module grouping options for more complex features) seems ideal to me.

That small core approach can give Drupal maximum flexibility, without sacrificing simplicity, exactly what Drupal needs to remain a strong competitor in fundamental network software structures.

And for my truly radical fantasy to share, I wish Drupal ran using Python. I just started programming in Python, and love the syntax and ability to properly name classes, etc.

Sharing that fantasy is 99% meant for a tiny humorous impact, of course.

However, just like a car requires the best parts for optimal performance, Drupal requires the best code language for that same result.

If Python configuration on a Web server becomes as easy as PHP, and if Python's popularity is sufficient to avoid alienating excessively large sections of our community, then some consideration on this front is reasonable.

February 2, 2011 - 13:04
Marco Sousa (not verified):

I think that in D8 we should have more documented release cycle.
Like: I central page with the roadmap (can be changed overtime, and funcionality can be removed or add new thing at any time)

In D8 I would like to the API involve like in D7. We can’t stop involving maintaining the advantage to other CMS, since all others CMS are changing the core to be more like Drupal.

The release cycle can't late more that two year, with Drupal 7 almost past the moment, we can't have other release soo long.. But now that is easy because all changes must have tests and can’t break anything.

We can create one new core module like “legacy compatibility” to support >60% D7 modules. (Will be slow that native module (D8), but is really useful when migrating one site that don’t have native modules, this will improve the migrate rate, and will reduce the complains that drupal release too often)

Continue to redesign the administrator pages, I think that we need radical changes one that.

Make more final product with more usability and functionality like Wordpress and Joomla to gain market share at the end users.

Some add to core modules:
1) We need a simple core module WYSIWYG extendable, with like 7 options.
2) Advanced user Profile
3) more fields
4) simple and extendable core view module
5) more core API for modules developers like (ctools, ..)
6) everything translatable
7) an simple and extendable print module.
8) An simple and extendable Fivestar field core module
9) An more acl permission and extendable (http://drupal.org/project/acl)
10) And simple and extendable Localization update module

So want the different from Drupal 7 (standard) installation and Worpress/Joomla? Is that we need other type of installation.
We have “Minimal” (for Drupal expert’s users)
We have “Standard” (for Drupal normal users)
We need “End Users /Completed / Full” (for Drupal End Users, demonstrations)

We can also release with profiles with in the core modules.
Like: Blog, News Page, Forums, Company.
So that way end users can have a ready demo page with demo configurations.

February 2, 2011 - 15:37
chx (not verified):

And what are you planning to work on? This kind of comment is really infuriating. "We can create one new core module like “legacy compatibility” to support >60% D7 modules" -- are you volunteering to try that? Have you tried doing such things before?

February 2, 2011 - 18:21
Marco Sousa (not verified):

I'm just giving thought's. I'm sorry if I infuriating you. If you think that isn't necessary, or too expensive work, just don't considere it.

February 2, 2011 - 20:26
Jimmy Berry (not verified):

Moving towards a Linux style development model sounds great. I hope that includes those in maintainers.txt gaining discretion over their areas as is the case with the Linux model and has been the biggest issue with core development.

February 2, 2011 - 18:26
Joe (not verified):

+1 for a roadmap of features for Drupal 8. Create a page that says Drupal 8 is going to have these features. That page and 0 critical issues will be the determination of when Drupal 8 will be ready. Of course this page needs to be set in stone and unchangeable otherwise features will keep getting added and thus D8 will be delayed even longer.

Another issue that bothers me is that it is hard to get into core development. I have made some modules and some of them are contrib on drupal.org, but I could never really get into core development because I couldn't figure out what they were doing. I had free time to help but since I couldn't figure out what they were doing I didn't help. It is partly my fault too since I have a very strict schedule which usually doesn't allow me to take part in sprints, irc, etc. I guess what I am saying is there needs to be some way for me to stay caught up with core developments so I can help. Maybe there is, but I have yet to find it.

February 2, 2011 - 19:05
catch (not verified):

Currently #drupal-contribute on irc is the easiest way - you can join that channel and find a lot of people working on things, then talk to them directly about possible areas to focus on. This is both a strength in that real time communication is great for keeping on top of core issues, but it's a weakness in that you don't see this unless you're on irc. From your comment it looks like getting on irc is tricky, but if you manage it and pop in saying you'd like to work on core, it's likely you'll get an answer pretty quick.

Off irc are the contributors block in the dashboard - that gives you links to patches needing review (of which there are hundreds, and they all need review!), as well as active bugs with no proposed fix yet, and there's also http://drupal.org/community-initiatives/drupal-core - which is an attempt to highlight particular issues that people feel are important - although it's manually updated so tends to get out of date regularly. There's also occasionally groups.drupal.org groups that get set up to discuss major changes, such as http://groups.drupal.org/butler or http://groups.drupal.org/fields-core

Then in the issue queue itself, there's issue tags, like performance or accessibility, as well as the novice tag which is supposed to include straightforward (not necessarily easy, but something that's not a major change mired in discussion) things for people to jump in on.

If you are finding it hard to jump in, one thing that is both really useful, and also gets you an overview pretty quickly, is going through the oldest issues in the queue (especially needs review or 'active') and trying to change the status - needs review can nearly always go to needs work or rtbc, active can go to 'needs more info' or add a suggested patch. This is doable with a limited amount of time each day, and in the process you might find something interesting to work on.

February 3, 2011 - 02:21
Vesa Palmu (not verified):

There are two common requests we hear from our customers: 1) Latest and greatest features 2) Long term support for their sites. Currently our options are a bit limited. We can always tell our customers that Drupal 8 is coming in less than two years and it will include a ton of new stuff. At the same time we are telling our customers that maximum support term for their Drupal version is less than 4 years. 4 years is still a pretty long time, but if you don't get your Drupal 7 site out right away it might be way less than 3 or 2 years.

Would it be a good idea to consider Ubuntu style LTS releases together with more frequent releases that could test latest and greatest improvements to Drupal core?

February 2, 2011 - 20:48
Alejandro Garza (not verified):

I would suggest we dedicate some times to have/improve our tools:

- I agree with catch's comment on the bot doing patch re-testing. Taking it a bit further, the test bot could also do a basic coding style check. More difficult (but very welcome) would be some sort of basic site performance testing, to make sure a patch doesn't degrade it.
- Webchick has talked about lowering entry barriers for those who don't even know what a patch is but have the will to collaborate testing out features, doing UX analysis, etc. So we could bring in those people providing more online tools: I'm thinking something like a widget that could accept a whole file (say: a changed X.module) and produce both a downloadable patch or a tar'd version with the patch applied.
- Why not have Dreditor right on D.O instead of a Greasemonkey script?

February 2, 2011 - 21:20
David Rothstein (not verified):

I want to see informal usability testing become part of the process of reviewing and committing patches.

Drupal 7's focus on usability was amazing - a great step in the right direction. But too many user interface changes either never underwent user testing or weren't tested adequately. This means that (a) some of the changes weren't as good as they could have been, and (b) we had lots of long debates in the issue queues, where everyone had an opinion but no one had actual facts or evidence that could move the discussion along.

Usability testing solves that. We don't always need formal tests in an official usability lab to make progress (although those are certainly good too!). All we need is for a few people to grab a spouse/partner/friend who isn't a Drupal expert, run them through a quick, scripted scenario, and write up the results. It's simple to do (usually much less effort than required to actually write the patches), and is easy for anyone, including new contributors, to help with. In issues where we did do it in Drupal 7 (such as this one), it was amazing to see how quickly people rallied around the actual evidence, identified trends, and came to a consensus on a final patch that was better than any of the specific solutions we had been debating before.

To do more of that in Drupal 8 mainly just requires willpower on the part of reviewers and committers to push for it when necessary. (And better use of the "Needs usability testing" issue tag, to keep track of issues that are at that stage.) However, we could also try to develop a better way to share the data that comes out of these informal tests. Having the test results live as comments in the issue queue isn't ideal, because although the test may have been designed to focus on the effect of a particular proposed change, often other unrelated issues wind up being uncovered along the way. We should store these writeups somewhere more accessible (and broken up into smaller chunks), where they can inform other issues later on.

February 2, 2011 - 22:15
sun (not verified):

My take:

  • We successfully introduced automated tests, but we didn't always require and enforce them. Only after reaching RC, commits became more strict and tests were required for every single change. Only a few people actually "lived" test-driven development; i.e., proving that a bug exists by adding tests first, only afterwards attempt to fix it.

    We can improve both on the strictness of requiring tests and on test-driven development.

  • We successfully mastered various highly complex and confusing issues and increased the level of detail of issue follow-ups, which allowed others to step into issues and review or help out more easily. Overall, we did a very good job in on-issue communication, coordination, and collaboration. However, especially for complex issues, the lack of a proper issue summary facility has been disturbing, both for reviewers and core maintainers. Closely related, we still have a full range of changes not being mentioned in the upgrade guide, which I explain with the current insanity of having to edit a single page containing a billion changes (and having to maintain them in case of follow-ups and rollbacks).

    We can improve the workflows around issues and changes.

  • We successfully gained a high momentum on important/problematic topics and APIs of Drupal core, and managed to pull plenty of contrib developers into core development for that sake. -- Interestingly, as the stats reveal, core contributions sky-rocketed only after the initial and originally scheduled code freeze. With a very high confidence, I'm alluding that to the fact that contributors saw a unique chance to fix problems in core, making the D7 release rock solid, and being able to use the improvements "shortly" afterwards; i.e., without having to wait years for the next major version. Everyone who participated in fixing those problems knows that a shorter release cycle would have left us with highly problematic and clearly broken/unusable functionality in an official Drupal core release, so shorter release cycles do not feel like the right answer. Instead, we should retain the high volume of contributions by allowing contributors to use HEAD for production sites (which implies to always do head2head updates in core).

    We can improve the availability of Drupal core during development, and thus, the amount of contributors as well as real world production testing of Drupal.

  • We successfully raised awareness of domain experts, and often (but not always) required them to sign off significant changes; after identifying and listing them in MAINTAINERS.txt, which overall heavily increased the confidence about the quality of changes. However, not all changes were signed off and especially for those, there have been some serious caveats afterwards. Furthermore, not all sub-system maintainers felt duly respected all of time, because some decisions were "overruled" by core maintainers (both by mistake and intentionally). Lastly, some sub-system maintainers did not always provide useful (i.e. cryptic) feedback, and for outsiders and new core contributors, the importance of "certain" issue comments has not always been clear.

    We can improve interaction and trust-level between core and sub-system maintainers, as well as between sub-system maintainers and contributors.

  • We successfully performed and completed an official sprint for a limited and clearly defined list of exceptions to go in after official code-freeze. While we focused on those action items, we still continued to improve other areas. Doing a "roadmap" like this upfront would probably not work. But near the mid and/or end of the development cycle, I think it was beneficial and helpful to have that defined list of acceptable exceptions. I guess it would have been even more helpful, if it had also contained non-feature items, such as "Fix the epic form/entity API integration and make it work", and if core maintainers would have published such a list more than once.

    We can improve the focus and productive output of contributors by introducing core maintainer "wishlists" (but without ignoring other contributions).

  • We successfully cleaned up lots of old, weird, and annoying code throughout Drupal core, indirectly leading to cleaner code, cleaner APIs, and better possibilities for integration and customization for contributed and custom modules. The semi-official "API clean-up" release cycle phase really helped to shift focus on these kind of issues. However, there's still a lot of mess everywhere throughout core, and by introducing new features and functionalities, we opened up new half-baked road works, leading to new weird, confusing, and annoying code. The already existing, old road works often complicated the implementation of new functionality, and the new road works of the new functionality are going to hinder and complicate future changes. To my knowledge, from all new functionality in D7, DBTNG is the only one that has been fully and completely implemented.

    We can improve our quality and increase performance of innovation by forgetting about a code freeze, and do a feature freeze instead.

February 2, 2011 - 23:43
good_man (not verified):

Re. your first point, I remember such a thing happened in CakePHP community where every ticket (i.e. issue) had to carry with it a simple-test and because very few really live the TDD, most of people simple ignored that and didn't submit tickets anymore. We have to take extra care of this point, you can't force people to write tests, otherwise they'll leave.

February 7, 2011 - 02:53
Mike Stewart (not verified):

I'd stress the most important thing would be to change to a six month release cycle. Two years is an eternity. and GIT is a game changer.

With the move to GIT, it is far easier to have a shorter release cycle because long-term projects/branches that can't be accomplished in six months are easy to manage and work on in parallel.

As I see it, the problem with two+ year release cycles is not only the perception, but reality that Drupal is moving slow. Technology changes. Many things important today will not even be relevant in 2-3 years.

Consider: there have been 13 releases of Ubuntu since it October 2005. There are numerous benefits to a time based release cycle. GIT handles any drawbacks.

I also believe the project will draw more help in from people who can make a three-six month time commitment vs. one-two years.

Shorter cycles are simply easier.
* Incremental changes are easier to manage.
* Feature/scope creep easier to handle.
* Less refactoring.
* Smaller contributor turnover/cycle.
* perception of being fresh
* important stuff is always being worked on
* and has the side-effect of reducing complexity and increasing understanding of a particular problem, memories work better, and fewer API gotchas.

February 2, 2011 - 23:59
yoroy (not verified):

D7UX was disruptive good for the focus on usability and thechanges it allowed us to make.
D7UX was disruptive bad because the timing (relatively late in the cycle) and actual kick-off drove away key core contributors. I'm confident this will be different for D8 as the awareness around importance of improving Drupal UX is there now from the start and a lot of less-impacting changes can be made to further improve Drupal usability.

Big architectural initiatives do not fit in a single issue in the queue. Has been said before, will be said again and we really need to do something about that. Anything that gets designed, decided, architected outside of the queue will be done all over again when it hits the queue. Either work outside the queue can be done in a way that it survives when entering the queue or the issue queue itself is improved so that it can handle larger initiatives.

String (and other) freezes that weren’t really freezes were sometimes frustrating. It’s fine to escalate the efforts a bit with deadlines but that might be handled better with committers announcing “I will focus on performance patches this week. Next week: UI strings, so start cooking up your patches!” Not sure what we really can do here though.

Issue tagging was a big win. It gave contributors that are interested in the more horizontal, all-over-core topics (ui strings, usability, performance) a good instrument to get an overview of what’s going on in their area of interest. We’ll want to improve that a bit, there’s lots of duplicate terms and we’re lacking a page that lists the main ones so that people can actually find out about these topics.

What I'd like to see happen for D8 is more room for design exploration. Too often the first solution to a given problem becomes the solution that gets implemented, without exploring alternatives first. That's a cultural change: don't be afraid to brainstorm, sketch out wild & weird ideas first before choosing a direction. And also a process thing, with more snapshot releases or specific core features getting temporarily moved to a more contrib like workflow with quicker iterations. I understand the new Git infra would make this easier to do, though I personally have a hard time grokking Git :)

February 3, 2011 - 01:46
catch (not verified):

Agree with all of this.

I think 'topic' (and/or large subsystem) sprints of a week or so is a really good idea, would need to be posted in advance as you say so that people can get their patches ready.

This wouldn't stop work on any other areas, but it means there's a chance to get particular queues into shape with focused effort.

Issue tagging, I hate tag clouds, but something like that would be useful, there may already be something that does this efficiently with solr.

Another thing the solr module does which is great is 'related posts' - it can use full text of the issue + comments for that, along with taxonomy terms etc. this would help to identify duplicate and complementary issues a lot faster. Will open redesign issue about that later.

February 3, 2011 - 02:40
Johan Dahlberg (not verified):

I think that the long release cycles can be a problem if Drupal should be able to resist the competition. I think you need to vary major and minor releases. Maybe three years is good for larger x.0 releases. While there maybe should be one or two bigger x.x releases between the major releases.

By having larger changes in the major releases you avoid the problems of backwards compability issues. Since there are plenty of time to migrate. But improving functionality and including modules into core could be practical in x.x releases.

To long time between bigger releases may not be very good if Drupal should be a strong competitor to the larger proprietary systems.

But most of all is probably the most important for the future in making Drupal attractive to big agencies and companies.

February 3, 2011 - 02:13
Dries:

Here is a possible approach. What if I selected 5 to 10 special initiatives (a la DBTNG, Field API, D7UX in Drupal 7). I'd pick these initiatives based on feedback from everyone in the community.

For each of those initiatives, I would appoint an official "initiative owner". Then, I schedule a weekly or bi-weekly meeting with the "initiative owner" -- other participants working on the initiative can participate in the meeting. We'd use that meeting to discuss progress, to define next steps, and to coordinate who is going to work on what.

This would provide more structure and communication for a select number of special initiatives, and could make it easier to tackle big and important problems. It provides a "partial roadmap", and the initiative owners would feel much more empowered because of the constant communication.

At the same time, it would not prevent the traditional grassroots development from happening.

Thoughts?

February 3, 2011 - 03:38
sun (not verified):

There's a risk here — of formalizing core development too much, into a closed and elite group and core contribution process.

I intentionally used the term "wishlist" earlier above, because the idea of a top-down decision scenario would at least make me leave into contrib. In other words: You and me and everyone else can happily publish a wishlist, but doing so does not mean that anyone is actually going to work on it.

I also don't think we've had communication issues concerning core maintainers — at least with Angie hanging around in IRC, all kind of issues were being actively discussed and communicated, regardless of whether being on the exceptional wishlist or not. It would certainly help to have more core maintainers (who have the final word) more often in IRC to ping about and discuss mission critical issues, but doing (bi-)weekly "meetings" (which I understand as conf calls?) sounds too much/formal/closed to me. If you meant IRC meetings, it would be more "inclusive" (i.e., everyone can poke, chime in, at any time, etc), and yeah, that's what we actually did for most of the major D7 topics.

Regarding "initiative owners", I'm not sure whether you meant mere project managers or actually project owners. The former, i.e., people trying to track the status and help to coordinate, sounds fine to me. But not the latter - they'd very quickly run into serious conflicts with sub-system maintainers, leaving us in a situation of having a core maintainer wishlist item, someone who owns it and tries to fulfill the wish, but domain experts who veto whatever is being done.

February 3, 2011 - 15:27
Dries:

There's a risk here — of formalizing core development too much, into a closed and elite group and core contribution process.

Transparency is important -- we should continue to do what we always have done. What I'm proposing is in addition to what we already do (e.g. IRC meetings, issue queues, etc). It's an additional meeting (probably a conference call) that increases our communication bandwidth and ability to scale.

Make no mistakes; I've had 'closed meetings' around all major initiatives. I participated in various UX and Field API sprints, I had private conversations with Larry about DBTNG, etc. In a way, my proposal formalizes the above and makes it more predictable. It creates more transparency as people would be more than welcome to sit in in those conference calls, we could record them, etc.

February 3, 2011 - 16:11
heyrocker (not verified):

I really like this idea as a good compromise around the issues of how our development works and the concerns that many members of the community have expressed over the last couple years. I particularly like the way it formalizes and opens up a part of the development process that has always gone on but has been opaque and invisible.

One concern I have had around the whole issue of changing our development process is that the move to git is already going to immensely change the way we work while at the same time opening up a wide variety of new possibilities. Going through a full release cycle will be an enormous learning experience from which I'm sure we can take many valuable lessons to be applied later.

I am niot one of the people concerned about release cycles. A too-short cycle has as much downside as a too long one, and honestly I dont think the problem with D7 was the length necessarily as much as it was the fact that the code freeze/slush seemed to go on forever with no end and numerous exceptions/extensions. I would rather see a stronger stance on milestones than any arbitrary statement of how long / short a cycle should be.

February 3, 2011 - 22:35
Dries:

Regarding "initiative owners", I'm not sure whether you meant mere project managers or actually project owners.

In my mind, an Initiative Owner (IO) is the person that I work with to make sure we are heading in the right direction. At the same time, it is also the person that contributors work with to make sure they have all their questions answered.

The IO should propose a mini-roadmap, define short and long term priorities, communicate about the progress, and herd the various contributors. The IO could also be the initiative's lead developer but that doesn't have to be the case.

An IO is a co-maintainer of sort, expect that he/she would not have commit rights. I'd also empower the IO to make decisions -- I'd be comfortable with that because we've been able to get alignment in our bi-weekly meetings. I'd understand the plans and sign-off on the short goals with the IO, and the IO makes all the micro-decisions. In case I need to help make a decision, it would be his or her task to weigh the pros and cons of the different approaches, and to help me and others to get consensus. Not unlike writing issue summaries.

It would also help me swallow and commit bigger patches because we've agreed on the approach ahead of time, and because the approach has been vetted and refined through the regular community process. It would avoid frustration because I can sign-of on a plan before implementation starts, and we can revise and update our plans at the bi-weekly meeting.

In the case that the IO is also the initiative's lead developer, the IO could be made responsible for combining and merging all the patches in his/her Git repository first. That would allow me to pull in bigger patches.

It is just an idea (and "IO" is a terrible name) but it feels like it could be an effective way to overcome some of our current growing pains.

February 3, 2011 - 16:41
catch (not verified):

I have mixed feelings on this.

On the one hand I agree that we already have various backchannels and informal structures, and if done right, formalising and documenting these makes things more open rather than less.

On the other, I'd be concerned how it would pan out - particularly in relation to which initiatives get chosen (subsytems? cross-subsystem topics? a mixture?), and who owns them.

In Drupal 7 there was an issue where we overhauled MAINTAINERS.txt - trying to remove people who no longer did any maintenance, and adding those who were.

This process showed that a number of people had completely disappeared after refactoring subsystems in Drupal 6, while many of the people in MAINTAINERS.txt for Drupal 7 appeared (almost) out of the blue (thinking of Damien and Bojhan for two examples in very different areas). We're not starting again from scratch with Drupal 8 and there may well be continuity, but it makes sense to leave things open to new people, and new initiatives, that might appear.

Also on a purely practical level, in terms of focused meetings on particular areas, whether that's with initiative owners or just in general, I'd like us to consider a new irc channel, possibly logged by Druplicon (unlike #drupal-contribute otherwise I'd say just use that), so there's a record for people who can't be online during conference calls due to timezone etc. - irc is as useful for the constantly rolling backscroll even when you're not paying attention, as it is for being able to talk to people in real time.

February 4, 2011 - 05:18
Dries:

"On the other, I'd be concerned how it would pan out - particularly in relation to which initiatives get chosen (subsytems? cross-subsystem topics? a mixture?), and who owns them."

As the project lead, I would pick the initiatives and the initiative owners -- similar to how I pick branch co-maintainers. I'd pick initiatives based on listening to developers, designers, usability experts, site builders, etc.

Candidate initiatives could be things like (1) configuration management, (2) HTML5, (3) more big usability improvements, (3) a better default install profile, (4) better web services support, etc. Each of those need a fair amount of brainstorming, planning and communication.

Initiatives would be temporary in nature, and so are the initiative owners. Their length may vary, and the initiative's leadership can change over time too. Initiatives should also come to an end because we completed the goals and it no longer needs special planning or high-bandwidth communication.

"In Drupal 7 there was an issue where we overhauled MAINTAINERS.txt - trying to remove people who no longer did any maintenance, and adding those who were."

The example initiatives above are not covered MAINTAINERS.txt. Some might be, of course, but I think that is fine.

"We're not starting again from scratch with Drupal 8 and there may well be continuity, but it makes sense to leave things open to new people, and new initiatives, that might appear."

Not everything has to be an "official initiative". As always, it would be perfectly OK to have "unofficial initiatives" like we've always had.

Anyway, I'm still thinking about all this. All the feedback is appreciated!

February 4, 2011 - 19:24
Larry Garfield (not verified):

I am also tentatively positive on this concept. (See my comments above about how lack of structure makes it impossible to "own" anything.) As you note above, there are already "blessed initiatives", they're just undocumented, not well understood, and not everyone knows what they are (including at times the people involved in them).

That sort of "explicit buy-in" would make it clearer what's going on. The number of initiatives may vary over time; we don't necessarily nee an exact list of all of them as soon as D8 opens, but we do need to know at any given time "what's the major actions going on, who's leading them, who do I go to to help out and/or ask questions?" A less ad-hoc approach can help make it clearer who is doing what where.

Regular status meetings, if I may use such a corporate term, can also be good to coordinate. The caveat is that since even those "Initiative Leads" are volunteers, their time will likely be uneven. (I know mine always is, at least.) That will pose logistical challenges that we'll have to work out.

One thing you touch on, that I think bears expanding upon, is that the "Initiative Owner" may or may not also be the lead developer for it. I can say with confidence from my own experience that architecting a solution, building a solution, and managing/coordinating people working on a solution are three very different skills and different people excel at different ones. Sometimes having one person doing most/all of it makes sense; for the first 2 years of DBTNG's life, for instance, I was the primary developer, architect, wrangler, and several other hats. Later on I stepped back and nudged and encouraged others to do more of the work, including bringing on a non-small team of co-maintainers for different drivers. These days, I mostly defer to Damien and Josh where Postgres is concerned, as the core system is fairly solid.

My point, I suppose, is that a given initiative may require more than one "owner", one focusing on architecture/development/code-herding and another on process/people/issue-herding. That's not a bad thing at all; I'm just noting that both of those are useful skills that we would need in such a structure that may not always be found in the same person. Witness the Git Migration, where Sam is the lead developer but we have other people managing the process around it. That team collectively "owns" that process, and we're on track to put CVS to the guillotine before the end of the month. (WOOHOO!)

Anyway, enough rambling. Tentative +1 to making more explicit and formal the process that we already kinda sorta use in fits and spurts. It's the fits and spurts that's the problem, and we can and should solve that part.

February 4, 2011 - 06:47
kika (not verified):

Following D7 post-mortem theme, i'd wish some things have been turned out more elegant way:

- spending time on core APIs what are superseded by better contrib alternatives: aggregator.module vs feeds.module
- the whole Epic Saga of Overlay
- adding complex UI features with no spec: dashboard
- getting jQuery UI to core with no proper vision/integration how to take advantantage of it, no pickup from UX, no theme architecture changes to support jQUI properly

One thing we need is more key contribs with one side of brain deep on code, other on UX.

February 3, 2011 - 15:05
Xano (not verified):

From a site builder's point of view the length of Drupal 7's release cycle was great. It meant Drupal was supported for four whole years!

From a developer's point of view the length of Drupal 7's release cycle was a pain. It took such a long time before we could use D7 and I completely forgot about most of the things I've worked on.

I think we should sit down and decide for how long we want to support Drupal releases. Based on that, we can decide a release's lifespan and how many releases we want to support at the same time. 3 releases times 2 years for 6 years of support? 2 releases for 2 years for 6 years of support?

February 3, 2011 - 16:02
Imre Gmelig Meijling (not verified):

This topic caught my attention as we discussed precisely this on drupal7meetup in Delft, 1 week ago. With several people present there with a business approach (which was partly the intent of the meetup), an interesting discussion unfolded.

I see many technical aspects to the development cycle discussion. Taking part in a Drupal development team, they are very important, for sure. The business aspects still tend to be overlooked a lot I think, which to me, is one of the aspects Drupal development teams still lack focus on. As most of us try to make a living with Drupal, the business aspects should therefore deserve their share of attention.

A continuous and symmetrical development cycle benefits us, because we can inform our clients about this. This way we can inform them ahead of time what the consequences are of a new release, both the benefits and downsides. A gradual update path gives potential Drupal clients more confidence. In other words: A constant update path is good for business.

While it common practise to support 2 Drupal versions at any given time, with Drupal 6 even so more businesses and websites run Drupal 6. So what happens when Drupal 8 comes out? The impact of not-supporting Drupal 6 will be far greater than with Drupal 5. I see no answer to this still.

I would be interested to see how everybody is dealing with these questions when it comes to their businesses. Anyone keen on sharing their thoughts from a business end?

February 3, 2011 - 17:55
twistor (not verified):

I'd like to see the components of core broken off into their own projects. This would compartmentalize them, making it much easier for the subsystem maintainers to know where their purview lies. It would also make it easier to have a legitimate chain of command rather than a tag team duo.

This would benefit users and developers. Each team would be responsible for their own documentation. Inciting a bit of competition here wouldn't hurt. The teams would also be responsible for vetting/introducing bug fixes and features before they got to the upper echelon. When users post a bug, they will have a more focus audience.

This would also allow for people of certain interests to focus on them. It would make it easier to identify people who have specific knowledge or skills since their contributions would be in a tangible location, possibly allowing them to contribute to a part of core.

The move to Git should help this a lot.

There is no reason that a person can't manage more than one component, the value in making each component its own project is a separation of concerns. It's simply a matter of the people at the top appointing people they trust to filter through the noise a bit.

It would also be nice because this process would mean a better definition of certain aspects of Drupal's architecture, as well as, separating the code more appropriately.

February 3, 2011 - 18:09
Berend de Boer (not verified):

The comparison with Ubuntu's release cycles is not helpful I think. Note that Ubuntu contains kernel + programs.

In Drupal we have kernel + modules.

Modules are frequently updated, and new modules and new features are released all the time, and people can take advantage of them without waiting for a new Drupal release.

A Drupal core release is a very disruptive event, and should therefore happen rather infrequently.

February 4, 2011 - 00:02
Jeff Eaton (not verified):

Dries,

This is a fantastic discussion to kick off, and I think the lack of solid post-mortems after each release is one of the things that has kept us from learning from as many mistakes as we could have over the years. There's a lot of great stuff here.

I think the Initiative Owners concept is a really important one; if there's a culture of solid communication between the IO's as well as to you, I think it could also help quite a bit to ensure that patches in one area of core don't go unnoticed by those working in others.

A few people have suggested ripping out nonessential modules and making them separate projects. Although I've long been one of the advocates of the #smallcore concept, I want to come out against the idea of cutting functionality and/or modules out of core in the D8 cycle. Separating the builder-facing-product from the coder-facing-framework is an important goal, but there's a lot of internal refactoring work to be done in core before that is possible. If we make structural changes in the Drupal project (like breaking core modules out into contrib projects) before the codebase is ready for it, the result will be a much weaker release for those who aren't already neck-deep in drush and custom makefiles.

February 4, 2011 - 07:38
Mukhsim (not verified):

"the result will be a much weaker release for those who aren't already neck-deep in drush and custom makefiles." - not necessarily. /profiles section of Drupal is next to useless now, it is time to fill it with meaningful installation profiles. One for a blog, one for a brochure website, one for an ecommerce, one for a forum etc. The beauty of it is that they are all independent of each other, thus can develop independently and in parallel. /includes and /modules should serve as a universal core that is necessary for 95% of websites to function (user, node, form api).

Currently core modules suffer from glacial times necessary for new features to come out. This does not have to be the case. The only way to address this is to go smallcore and set modules free, while providing site builders with useful installation profiles. The fact that a module makes it to an installation profile does not mean that it will not see a light of day until next Drupal release. People should be free to update modules with newer versions.

Six months release schedule will incorporate the latest stable releases of those modules and add new modules to installation profiles, if necessary, all the while keeping things under the hood (/includes and /modules) stable. And it can be done without much effort, D7 only needs a handful of modules to run, everything else cam be moved to installation profiles.

Mukhsim.

February 5, 2011 - 04:04
Jeff Eaton (not verified):

"/profiles section of Drupal is next to useless now, it is time to fill it with meaningful installation profiles. One for a blog, one for a brochure website, one for an ecommerce, one for a forum etc."

There's nothing preventing that from happening today: Install profiles have been with us for half a decade now, and I can count on one hand the number of reasonably well-known install profiles that exist in the Drupal world. The problem, at least in my opinion, is the complexity and difficulty of managing the creation, testing, and support of an actual install profile.

Part of that will never go away -- building a product-oriented profile for site builders and users will always involve more labor-intensive polishing than building an API for developers. But we can also simplify the process by making it easier to build profiles. That can be done without tearing pieces out of core at all. Gutting core first, and hoping that install profiles will follow, is rolling the dice with Drupal's future.

The problem, of course, is that if we don't force the transition somehow, we'll just continue on our present course with no incentive for change. The problem we saw in the D7 branch was that most of the people with lots of experience building profiles and distros were busy supporting those profiles and distros for D6, and the people most active in core development for D7 had never built a real distro-style profile, only tested the Install Profile API to make sure it was still working as documented.

This is why I've begun to advocate a project to flesh out Drupal's own default install profile, focusing it on a more specific target audience and treating it as both a useful tool for end-user site builders, a useful example of how to build sites for more advanced site builders, and a way to exercise our core APIs to ensure that profile building is easier. It's the only way -- IMO, at least -- to ensure that making feature-rich distros remains a core priority without punishing those who use Drupal in the interim.

February 5, 2011 - 18:19
Dries:

Yes and no.

I agree that creating and maintaining a distribution (or installation profile) is labor-intensive. However, the fact that it is labor-intensive isn't a problem if there is an economic incentive to invest that level of effort. People have done crazier things; like digging for gold with a pickaxe. ;)

The real problem is that in most cases a Drupal distribution is nothing but a not so cost-effective lead generation tool.

The most successful distributions are those that target the enterprise because the financial reward can actually exceed the engineering investment. Which also explains why we don't have a successful "blogging" distribution for Drupal yet.

Lowering the technical barrier to building and maintaining a Drupal distribution, lowers the investment cost so it will have a positive effect -- we should certainly strive for that. The question is whether it will move the needle. The lack of a viable business model for distributions is likely to be the real problem -- at least if our goal is to see a thousand distributions bloom.

February 5, 2011 - 23:40
Jeff Eaton (not verified):

"However, the fact that it is labor-intensive isn't a problem if there is an economic incentive to invest that level of effort... The real problem is that in most cases a Drupal distribution is nothing but a not so cost-effective lead generation tool."

The same argument, though, applies to Drupal modules and Drupal themes. There is no direct path to monetization for module developers, and yet many people write them. Drupal, today, is used to solve many important but unprofitable challenges. We are an open source project, after all. :-)

"Lowering the technical barrier to building and maintaining a Drupal distribution, lowers the investment cost so it will have a positive effect -- we should certainly strive for that. The question is whether it will move the needle. The lack of a viable business model for distributions is likely to be the real problem -- at least if our goal is to see a thousand distributions bloom."

To a large extent, the technical challenges for profile and distro builders are the same ones facing us in the area of deployment and staging. Solving those problems effectively will reduce the technical hurdles for most profiles considerably, to the point that building one won't require an independently funded team of developers.

When that happens, the question of monetization will be moot just as it is for modules. Some distros targeting lucrative markets will be profitable (as you suggested) and others will not be, just as we've always seen with Drupal work.

I think we do see eye to eye on a lot of the inherent challenges, but like economics discussions that revolve around supply solutions vs. demand solutions, we're approaching it from different directions. You're talking about making expensive work more profitable and I'm talking about making rewarding work less expensive.

February 6, 2011 - 19:02
Dries:

You're talking about making expensive work more profitable and I'm talking about making rewarding work less expensive.

I don't think that is a fair statement. Both are really important. I think it is critical that Drupal wins the hearts and minds of developers -- something that will take a lot of work.

It just feels to me that for most developers, writing a module is more rewarding than writing an install profile. Not because of the technical challenges, but because of the inherent nature.

February 7, 2011 - 02:53
Larry Garfield (not verified):

I don't know that "see a thousand distributions bloom" is necessarily our target goal. Part of the question there is what the purpose of an install profile is.

Is an install profile a tool for building OpenAtrium, Managing News, Drupal Commons, and other "enclosed applications that oh yeah you can extend", or is it to make it easier to build the first-third of a site, so that you or the consultant you hire can have an easier time (and more available budget) to build that next two thirds?

Those are very different target tasks, and which one we want to support will dictate what we do with install profiles. Eg, the more functionality that is tied up in an install profile rather than being part of a general-use module, the harder the second task will be. The second task is much better suited to Features-style pluggable packages. While one can build a profile/distribution system around feature-style pluggable packages, there are other ways to do so as well.

IMO the focus should not be on making install profiles more powerful, as that only benefits the closed-app market. The focus should be on making functionality chunks easier to mix and match a la features or something like it; that benefits both groups above, and makes an "install profile" really just a meta-package for a collection of features. That's much more flexible.

February 6, 2011 - 19:05
catch (not verified):

This is a great way to look at it, and you explain the tension well.

It reminds me of This image from this development seed blog post about features.

Conceptually the second image is a nice representation of the kind of thing you describe.

The install profile itself, is the 'fourth layer', not pictured in those images, - which depending on how we do it, could either add to the current mish-mash of the former, or be a nice thin layer on top of the second.

February 7, 2011 - 07:46
Cary Gordon (not verified):

My perception is that each time we try to compare Drupal to other FLOSS projects, we get in trouble. Certainly we have things in common with other projects or aspects of those projects, but overall, we are unique.

I do not think that the evolutionary path of Drupal has brought us to a point where shorter development cycles or an agile approach are going to be possible, let alone practical. I do think that a two years cycle is doable if we can do a better job on the disciplinary points mentioned earlier.

Having started my project involvement on the documentation side, I strongly agree that documentation needs to be at least as fundamental as testing has become, and to be clear, I think that we need more of both. The idea that a good, or perhaps simply deranged Samaritan is going to come along and figure out how to explain your work is delusional. What winds up happening is an aggravated Samaritan winds up documenting your work, without your input, and while they do the best they can, it is woefully short of what it should be.

I think that Ariane, Jennifer and everyone else who works on docs are saints, and they need all the help we can give them.

I think that having framework and product release managers for D8 is a solid idea. Angie and Dries had a sisyphian task on D7, and any division of labor that is both workable and sensible should be considered. The thread on Eaton's site is a great discussion on that topic.

For my own part, I want to see internationalization become a fundamental piece of Drupal 8. It should certainly be a consideration in designing the next framework and fully usable at the product level. Why? Because Drupal is in some ways one of the most international of projects, and while bringing in locale was a crucial move towards empowering sitebuilders working in languages other than English, we need to go further and elevate the folks working in multiple languages, as well.

February 4, 2011 - 20:53
dasjo (not verified):

i would like to see peripheral subsystems as separate modules with a better separation of drupal the framework and drupal the product as the lead drupal distribution. we would make even more use of unit- and integration tests in order to keep the quality high. from my point of view, in combination with the successful git migration, this would allow even more participation in the core development process while putting the focus on cleaner roadmaps and better coordination of the development efforts.

February 5, 2011 - 08:58
Sean Burlington (not verified):

The main change I'd like to see in Drupal developemnt is a reduction the time from contribution to release.

I think this would really help encourage more contribution from professional developers - working on client time.

I've supplied the odd patch to core - one was even accepted - but that was 3 clients ago - so I can't really contribute to Drupal in billable time.

As long as Drupal has no feature enhancements within a version, doesn't retain backwards compatibility and has code freeze for months before a release - there is no time when I can contribute functionality to core that my current client might benefit from.

What I'd be especially keen to see is interim releases of Drupal within the 2 year cycle of main releases.

These could add functionality where backwards compatible, but most importantly they could add API functions that help with install profiles and update hooks.

install_profile_api is a good module - much of it should be in core - and could be added gradually.

February 8, 2011 - 12:17
Miha Chenkov (not verified):

I absolutelly agree with Sean Burlington. Other frameforks (e.g. Yii, Kohana, etc.) have a lot of new enhancements and features after each new minor release. This way allows to have modern core.

February 8, 2011 - 21:59
bojanz (not verified):

2 years is 2 centuries in internet time. We are moving too slow.
I think that any release schedule that takes more than a year is too long.
In any case, there are real ways to speed up core development, which can be quite slow at times.
Many issues become "stuck" and it takes a herculean effort to make them move forward again. We now why they get stuck. Everyone's opinion counts the same (it can't), maintainers have almost no real power (they should), too much depends on two overworked core committers, and it's not even their full time job to commit (leading to long RTBC queues, and issues awaiting feedback, which I'm sure added at least weeks to the D7 release date). Even I (a very casual and recent core contributor) could see and feel this.

So, any way of dividing responsibility to more than two people is a good thing and definitely necessary. With git and git branches it doesn't matter whether they have commit access, if Dries can trust them and the general direction of the code and just merge their branches...

We can't schedule much code writing in a open source project, but we can eliminate many bottlenecks that attribute heavily to this problem.

And as a I side note, if it takes paying a couple of people to commit, lead, and channel discussion, to have a release months earlier, then I'm sure that's a cost that we (the community) can gladly support, since nobody can argue there's very real money around Drupal.

None of the above is a critic against Angie. She is several levels above awesome, and we need 10 of her. If we can't clone her, then we can at least get every minute of her working time, instead of 3, 4, or 5 hours a day. Same for every other leader.

February 9, 2011 - 01:56
PWolanin (not verified):

I think one way to get faster feature releases is to think about the discussed framework/application split.

I'd agree that the 7 cycle was vastly too long. But breaking all the underlying APIs after ~18 months causes a lot of pain for existing sites and ongoing projects. Can we envision a revision of the "Application" layer nearly yearly and a longer cycle (2+ years) for the "framework" layer?

February 10, 2011 - 18:09
effulgentsia (not verified):

Wow. A lot of great discussion above. Here's some of my thoughts:

  • Testbot rocked! When I first heard about the initiative during D5 and D6 days, I was skeptical. I thought it would take forever to get halfway decent enough test coverage to be worth it. It did take a while, but not forever, and where we are now with it is purely awesome. It's exciting to now get to benefit from it from the beginning of D8 development.
     

    We now need to get a test framework for the JavaScript code. And we also need automated performance testing.

  • The focus on usability and accessibility rocked! In addition to resulting in a better Drupal, I feel our community is now improved due to the addition of more UI and accessibility experts, and a development culture that's increasingly paying more attention to usability and accessibility concerns.
     

    We still have improvements to make in how we best collaborate within a larger and more diverse community. For example, managing UI issues in the issue queue has proven harder than managing code issues in the issue queue. But I have confidence we'll figure this out.

    Also, we now need to bring focus to issues around mobile devices and internationalization.

  • webchick rocked! If you can find one or two D8 maintainers who can do what she did while also keeping a day job, awesome!
     

    Otherwise, I think it would be good for the D8 maintainers to be funded positions, so that it can be those people's day jobs. I know there's lots of challenges with opening up some positions as being funded, such as who funds it, and what unwanted control do they actually exert or are perceived as exerting. And also, fairness and motivation issues around some people busting their ass on volunteer time while others get paid. But hey, the Drupal platform and community is growing, and I don't think there's any question that if the money can be raised, and if the challenges can be addressed, that having certain key roles being done by people who can devote full attention to it, will benefit the entire project and community. And even in D7, despite you and webchick being totally awesome, we still had many patches sitting in the RTBC queue for months, extending the duration of the development cycle, and demotivating contributors.

  • We had some excellent examples of companies sponsoring sprints or bursts of activity when things got stuck. Some of this happened early in the cycle. Some happened during alpha, such as when Clarity (examiner.com) sponsored both in-house staff and opened up bounties for some of the thorny issues. And some more happened during beta, such as when Acquia sponsored first ksenzee and then me to each spend 3 weeks full time on the critical queue. All of these bursts of focused attention were critical to getting things unstuck.
     

    I hope we see more of that during D8. We'll probably have a lot of things we want to accomplish in 2 years or less. A lot of that will be do-able by many people each working on hobby time. But some will require a burst of a few people working full time. I hope we can more quickly detect when the latter is needed, and find a way to make it happen.

  • There were some notable examples of excellent mentorship. In particular, I want to thank sun, moshe, and webchick, who I've personally witnessed providing countless amounts of encouragement, direction, and constructive reviews to people who will no doubt become prominent D8 and D9 contributors. There's no doubt many other people who deserve mention here.
     

    But we need more. There's many forms in which mentorship can take place, and many of those forms are hard to measure. An easy one to measure, however, is how long issues sit in the "needs review" queue. If you're already an expert and respected contributor, simultaneously working on dozens of issues in the queue at a time, then having an issue wait a long time for review isn't so bad, because chances are that when you have time to contribute, at least one of your issues has gotten a review that you can follow-up on, and if not, you know who to pester, and can likely get their attention. However, if you're a relatively new contributor, starting with just a couple issues you're working on, then having those wait for months for review is massively demoralizing. I'd love to hear ideas from people about how we can better encourage more mentorship.

  • We have a lot to think about with respect to GIT. For example, I find that getting an issue to RTBC is most helpful when the issue is as small as possible, because that makes it easier to keep everyone focused, build consensus, and provide detailed reviews. However, Dries, you have stated that you prefer reviewing one big patch rather than the equivalent amount of code spread across 20 small patches. So, GIT presents us with an opportunity to have a workflow where small patches get committed into a "feature branch" or "feature repository", and then an entire feature can be staged for review by a core maintainer. But, how do we organize features, and whom do we empower to commit to feature branches or feature repositories? Do we split by subsystem, so that subsystem maintainers can commit ready patches that are localized to their subsystem? Do we split by large initiatives, with initiative owners having commit rights to the corresponding branch/repository? Or perhaps both? Done right, this can greatly help prevent the huge RTBC backlogs we saw in D7. But, it also has its own problems it might introduce that we won't fully come to terms with until we've done it for a full development cycle.
  • Following up on comments above that identified a need for both relatively long times between major releases, but shorter times between not-just-bug-fixes minor releases, I think we should evolve to a major.minor.fix versioning scheme and workflow. We can't do this for D7, but we can evaluate the feasibility and desirability of doing it for D8, such that when D9 development starts, we can also be backporting more significant work into an 8.1.x. I agree with catch that we can't easily split major/minor by framework/application, or by BC-breaking/non-BC-breaking. Given Drupal's architecture of hook_EVERYTHING_alter(), even very small changes can be BC-breaking, which is why we backport so little to already released major versions, and I don't think Drupal should move away from a hook_EVERYTHING_alter() architecture. But, I think we can split things by majorly-disruptive/minorly-disruptive, and allow potentially BC-breaking, but useful, minorly-disruptive improvements into 8.1, while reserving 8.0.1 strictly for critical or non-BC-breaking bug fixes. As Paul points out, modules and themes can then list "<8.1" as their dependency when released for 8.0 (as in, "I don't know if this module or theme is compatible with 8.1, since 8.1 isn't released yet."), and when 8.1 is released, then the module or theme maintainer can review the core changes, test their module or theme against it, and update the .info file to "<8.2" when sufficiently confident that it's compatible with 8.1. Site builders can then upgrade to 8.1 when all of the modules and themes they're using are compatible with it.
February 13, 2011 - 19:27
catch (not verified):

I'm surprised it didn't come up in these comments earlier, but branch maintainer being a funded position is something that's been discussed on and off in irc for a while, and if there's any chance of that happening the discussion needs to happen sooner rather than later.

I like the fact that the majority of funded contributions to core are more or less indirect, and when it's direct, it's usually temporary - since there is a massive difference between paying someone to fix a specific issue, that results in a core patch, compared to paying someone to write core patches in general, and IMO the current balance is pretty good (althoughf Larry's Drupal ladder of engagement shows that this kind of funding is likely much less common than others.

However I think with the branch maintainer role, it would be good to try to break this model (as happened with Gabor for the last six months of Drupal 6 - paid full time as Acquia's first employee).

The branch maintainer is a single (well, dual or triple) point of failure/bottleneck - which for example means that at least me and some other core contributors not on US time had some very, very long nights trying to be online around the same time as webchick's core time - which often started at 2-3am in Europe - while also competing against other people who wanted to discuss their own patches as well. Angie did her best to accomodate all the various time zones, but doing 4-8 hours/day of core work, on top of 40 hours/week for a day job, for three years, is pretty unsustainable.

It's also much easier for everyone that's not the branch maintainer to take a few days or weeks off core without it inconveniencing other people - since while it might hold up the things they're working on or people who rely on them for advice, it doesn't create a system-wide bottleneck.

So funding someone so they are not doing branch maintainership on top of a 40 hour week, rather than creating ill-feeling about who's getting paid, would I think would be a massive relief to people who are trying to juggle multiple RTBC patches - and get them reviewed, re-reviewed, re-rolled etc. - which is often just a sheer quantity issue that requires a lot of dedicated time to keep up with consistently. Momentum in the RTBC queue (momentum rather than actual commits) really does affect overall morale with core developers - the more issues I have RTBC at the same time, the more likely they are to go stale, the more reluctant I am to fill the RTBC queue with even more patches etc. etc.

I don't have great ideas for keeping the reviews queue down - at various points me and others have tried to go through the whole thing, or a slice of the whole thing, to cut it down, but that's becoming more or less impossible with the current quantity of patches unless you had a few people doing this more or less constantly. As mentioned above, I think the main priority should be keeping the criticals queue close to 0, including short code freezes to enforce this if necessary, that might help focus efforts on the reviews queue as well.

On version numbering, with 7.x we could potentially save some bigger backports from Drupal 8 for 7.5 and 7.10 - and try to co-ordinate so that these aren't security releases. this would be hard but it would allow for some of the same expectations as 7.0.1 even if it's just a hack until we can sort something better out for D8.

February 14, 2011 - 14:38
Larry Garfield (not verified):

The branch maintainer is a single (well, dual or triple) point of failure/bottleneck...

That is in and of itself a critical bug in our process, independent of whether or not that bottleneck is being paid.

February 14, 2011 - 21:55

Add new comment

© 1999-2014 Dries Buytaert Creative Commons Attribution-NonCommercial-ShareAlike 3.0 License.
Drupal is a Registered Trademark of Dries Buytaert.