Backward compatibility
The issue of breaking backward compatibility comes up every once in a while -- more and more in the form of a heated discussion.
When I first released Drupal, I chose not to preserve backward compatibility, because I was mainly interested in getting the technology right. Preserving backward compatibility often requires that you drag historical baggage along, and in interpreted languages like PHP, this comes at a significant performance cost. So it was decided that we can break people's code, but not people's data. Our mission was to make Drupal fast, small, clean and on the bleeding-edge of technology. In the early days I focused, completely and utterly, on the aesthetics of Drupal's code. I spent days trying to do something better, with fewer lines of code and more elegant than elsewhere. And with me, many others.
It was the right thing to do. Over the years, we've seen a lot of innovations happen that would not likely have happened while preserving backward compatibility (the node system being one of the most prominent examples). Developers always had free reign to implement their ideas in the best possible way. It is something that Drupal has in its favor compared to many other content management systems. It's been interesting to see how Drupal has spread and how it has grown to be more flexible and cover more niches than many other systems. If anything, this needs to be attributed to the fact that we haven't cared much about backward compatibility, and our single-mindedness to get the technology right.
Of course, breaking backward compatibility has its disadvantages. Expensive upgrade paths and frustrations due to lack of up-to-date modules being the examples that immediately come to mind. And for many people, this matters a lot.
Unfortunately, there is no right or wrong answer here: there are both advantages and disadvantages to backward compatibility. As a result, there will always be a tension between the need for hassle-free upgrades and the desire to have fast, cruft-free code with clean and flexible APIs. At the end of the day, we can't make everybody happy and it is very important that you realize that.
Most people, however, just worry about how they are going to upgrade their Drupal sites by next Friday, at the latest. As Drupal grows, these users become increasingly verbose about the issue of backward compatibility, and as a result, it becomes harder and harder to maintain our original core values. For many of us (including myself), this is hard to accept. It is hard to accept not because we don't care about people's problems, but because we realize that the only viable long-term strategy is to focus exclusively on getting the technology right. The only way to stay competitive, is to have the best product. It doesn't get any more complex than that. If you start dragging baggage along, your product will, eventually, be replaced by something that offers the same functionality but without the baggage. Something that is faster, smaller, cleaner and on the bleeding-edge of technology. Unfortunately, only those with a long-term interest or vision tend to care about this. End-users with sites to upgrade are challenged by a different set of problems, like a Friday night deadline. They are extremely short-sighted, and rightly so. Remember that both sides in this argument have valid points.
Given the fact that Drupal's main strengths have always been the agility with which it can respond to the ever-changing landscape of web development, and the almost infinite amount of flexibility and expansion it affords developers, I feel that preserving the ability to constantly innovate is of higher importance, at the present time at least, than preserving backward compatibility. None of us would be here now using Drupal were it not for this core value, and I strongly believe that it is fundamental to our future. It always has been.
But what to do if many of your users slowly force you to change one of your core values? It seems inevitable that sooner than later, we will have to be a lot more careful about breaking peoples' code. And when that happens, I fear that this will be the end of Drupal as we have come to know it. Probably not immediately, maybe not even for several years, but eventually Drupal will be surpassed by technology that can respond more quickly to change. Maybe that is bound to happen anyway, and when it does, having made as many people happy as possible (even at your own expense), is the only thing that really mattered?
Well, you do not wish to end up as Microsoft, having to tote around loads of ancient stuff. I think that if you port back new features and bugfixes to older releases, you'll be fine. Users who do not wish to advance at the fast pace the bleeding edge sets, can go along with the maintained older releases. I take FreeBSD as an example here, where for a long time stuff was ported back to 4.x from 5.y and even to 3.z IIRC. Mainly bugfixes, but sometimes also features. Things that won't break code based on older releases, but that would be nice to have nontheless. IMO, that is what you have branch maintainers for.
But in the end, you want all your users to upgrade, but perhaps at their own pace.
Just keep it lean an mean at the front, and you won't get dragged down by the balast you've acquired.
May 17, 2006 - 09:23Currently, stable Drupal branches are meant exclusively for bug fixing, and so we have a policy of not backporting new features. There are, however, unofficial patches that backport important functionality. Changing our policy to allow important changes to be backported would certainly reduce some people's urge to upgrade. It is an idea worth exploring.
It does, however, make future upgrades more complex. Drupal would have to support multiple upgrade paths as you'll end up upgrading from vastly different versions. It's a technical problem we'd have to solve first.
May 17, 2006 - 10:13I'd rather see a shorter release cycle as we are trying to have it for the next version. This also solves other problems while not introducing new ones.
May 17, 2006 - 13:15To upgrade or not to upgrade? That is the question. One that people seem to forget to ask themselves in the first place.
No one, or nothing, forces you to upgrade. Drupal 4.6 has been a very good product. So why do you suddenly need Drupal 4.7? Add to that one of my favorite one-liners «never fix something that ain't broken» and you have half of the solution.
We have maintained security patches for older versions, and will continue doing so, I am sure. Security is not the reason you have to upgrade.
Is it the new shiny features you want? Then you are most probably looking at upgrading with the wrong mentality. Upgrading is not something "for free". You should think of it as "improving your site". Or as "adding new features". Both require resources. Everyone knows that it takes coding, resources, planning and often a budget.
So, personally I think that supporting backward compatibility is bad. It will only facilitate those people that want the lastest-greatest, but cannot accept the fact that it requires resources.
Another point might be that you choose to develop a new site on Drupal 4.7, but find that modules are not yet ready for it. In that case, you will find that developing such a site requires more resources then developing on Drupal 4.6, where required modules are available. This too is a mentality thing: just accept that fact, and calculate that into your budget.
We could look at better management for older branches, maybe. But with the amount of people around, I expect that to happen anyway. I expect that Drupal 4.6 will continue being used for years. At least by those smart enough not to «fix something that ain't broken».
May 17, 2006 - 09:47Interestin times. Lately, before this latest thread broken off, I did write some insights on Drupal upgrades for Drupal.hu (in Hungarian), and I have choosen to start with suggesting people to first consider if they need that upgrade at all. I did have a site for which I myself will not consider an upgrade for months to come (from 4.6). It just has so many custom stuff, and it works nicely as it is.
I deeply echo the sentiment that in the FOSS world, not everything is for free. People just got too accustomed to this great freeness :) Even if something has no direct cost associated, time or human resources need to be taken into account. Quite a few of the great new features are possible in 4.7 because backwards compatibility was broken, and improved stuff was included. Now if people like these new features, and they intend to upgrade, they need to accept that there were some compromises in backward compatibility to make the new shiny stuff happen.
May 17, 2006 - 13:58I still believe (sorry, no hard data) that the total cost of ownership for a Drupal based site is still much lower than for comparable systems even if one considers that you need to put some work into upgrading modules and themes.
I have clients that stick with Drupal for several versions now and they don't seem to mind the occassional consulting hours for upgrading custom modules. Maybe my clients are smarter than other people's clients. ;)
Also I believe that the Drupal developers should not allow their work to be wreestled away from them by anybody for whatever reason. Such ongoing attempts must be strongly opposed. As you write, the developer centric approach is what makes Drupal great and any change would probably make it more like some systems people like to call the competition. ;)
May 17, 2006 - 10:55But what to do if many of your users slowly force you to change one of your core values?
As I listen to the conversations about backwards compatibility, and listen to what you lay out in your blog, this is the line that resonates the most.
IMO, it is unrealistic to expect you to alter your core values to accomodate somebody else's business needs. Does the lack of backward compatibility mean that Drupal might not be the choice for every client? Absolutely. But, as a consultant, part of my job involves matching a tool with the client's needs.
FWIW, I strongly urge you to stick to your core values. Those values have helped create a body of code that permits a growing number of people to make a living.
May 17, 2006 - 11:13As a developer, the idea that the technology comes first certainly resonates. Being held back from doing the fastest, most cruft-free code by the need to accommodate your users is painful. Still, the argument begs for a devil's advocate.
Let's be a little philosophical for a moment. What is Drupal? When you were developing it, and no one was using it, clearly it was the code, and your ideas for its future. Most importantly, it was yours.
Is that what it is, now? Now that large numbers of other people have contributed to its development? Now that thousands of people are using it? Now that some piece of the the success of lots of other good work rests on it? Is Drupal still just code, to be driven by coders' values? Is it even yours?
From a practical perspective, do you think that permission to break backwards compatibility will be sufficient to prevent Drupal from being overtaken by someone who has no user base, who is unrestrained by pushback from other developers, and who can start from scratch with the latest and greatest technology?
Is the "baggage" of which you speak actually Drupal itself?
May 17, 2006 - 12:44I think that you have chosen the correct approach, so far. I think you can maintain your core values--for the most part--by providing better upgrade tools.
This will allow you to continue to innovate and change code, but helps those with many Drupal sites to maintain.
The 4.5 to 4.6 upgrade, especially for the DB, was something only a Drupal developer could love. I am a very experienced Web site manager with a degree in computer science, and was unable to correctly upgrade 5 sites. I had to hire someone to fix the mess.
This was an *upgrade tool* problem, not a philosophical problem about changing code too much. In the end, the problems were relatively simple--better tools would have made the upgrade much less painful and less costly.
I agree completely with the "change the code, not the data" approach. For the sites I maintain, data integrity is more important than code integrity. Having said that, there is a point at which too many code changes too fast becomes quite difficult. But I have not seen that yet.
Andrew
May 17, 2006 - 13:33Completely agree that good code is more important than backward compatibility.
There are a few things we could do to soften the impact that has on the upgrade process...
Frequency of updates
I'd say every 4 months is just right, enough time to develop and test new features, and spread the cost and labour of updating out across the year.
Better defined Alpha, Beta, RC Stages
May 17, 2006 - 13:39Make it clear what each of these stages mean for Drupal, so projects can plan which version to develop for. For example:
Alpha - Existing APIs frozen, new APIs may be added.
Beta - Pure bug fixing, no new features or API changes (unless unavoidable).
Release Candidate - No critical bugs, test and fix 'normal' level bugs.
In general, I think users are not as concerned with backward compatibility as they are with how long a particular version of Drupal will be supported and maintained. Also, when they are ready to finally upgrade from 4.5 to either 4.6 or 4.7...can it be done easily?
The Ubuntu Linux distribution is a well respected distribution among the Linux hobbyists...but it has had a difficult time being taken by more serious folks due to the same issue facing Drupal. To ease users' concerns, Ubuntu has recently assured users they'll support new releases for at least three years. What assurance do Drupal users have that they won't be forced to upgrade from 4.7 in 18 months?
I likely know the answers the these questions and am comfortable with Drupal. However, the end-users really want things well defined. The need for warm fuzzies and attention to politics evidently comes with the territory of managing a successful project.
May 17, 2006 - 14:01For myself, keeping my data intact is fine. And if you're a php programmer at all the code upgrade instructions that is ALWAYS provided make updating my modules pretty straightforward as long as the original code was reasonably structured.
I think I understand the desire to stay on the edge...I can't picture, for instance, adding AJAX functionality to a Drupal 4.4 installation (but them Javascript isn't my forte anyway). But if you created the website you intended to in the first place, I can't see a great need to update to the latest and greatest.
May 17, 2006 - 14:47Well, I'm somewhat in the minority.
What I tend to prefer is backwards compatibility to a point. I think having backward compatibility forever is extremely wasteful.
That said, I often get the feeling that people are too willing to break the API because of this policy. In a lot of the cases, backward compatible functions can be left in and marked deprecated, given a specific lifetime, and then removed.
I do feel that one emphasis of focus is lost here: one thing Drupal really demands is an active group of contrib developers. And we have that, but contrib developers will burn out by having to update module after module after module because of constant API evolution.
My thoughts are to maintain backward compatibility in minor versions [where possible; I don't advocate keeping it where it's just not feasible. see: formsapi]. That said, making an effort to preserve a deprecated API at least for a version can be a benefit. And every major version rev can kiss backward compatibility good-bye. Few people would expect it.
Perhaps that's my major issue. When I see a minor version increase, I tend to assume that the versions will be relatively compatible, and they are not. It can be distressing, and is extremely confusing to users coming in.
As we move forward, Drupal finally actually knows what its own version number is. This is a major step; I can see where modules will want to keep code with dual compatibility, especially since Drupal's philosophy of API breakage means that many of those entrenched 4.6 sites will never ever upgrade.
May 17, 2006 - 16:28Merlin, you have an almost unique point of view. You developed your great but huge modules along 4.7 and kept them up-to-date while 4.7 API was real fluid. This surely was very tedious.
I think that if someone wants to provide legacy API, he is free to do so in a contrib module. That is not a task for core.
Dries, do not let Drupal be dragged down by whiners. They think that open source is a free lunch, but as we know, there is no such thing.
May 22, 2006 - 06:04I'm a consultant who's a Drupal newbie. Unfortunately, as it turns out, I didn't find anything suggesting that there would not be an upgrade path out of flexinode into CCK for the data, or (yes I'm dreaming) into Views.
I know I'm a minority here, but I'll speak anyway. At least a bit more investment in upgrade paths for well-used functionality would help the community in my view. It would make it easier for me to recommend Drupal, and more likely that my clients will upgrade, making it more likely their requests for new functionality get built into HEAD rather than for old versions like 4.6.
May 17, 2006 - 16:57The 4.7 release has made updates of contributed modules much more painless than it as in the past. I am quite sure that there will be an upgrade path from flexinode to cck once cck is actually released.
May 17, 2006 - 21:31Somewhere else, you suggested that freezing APIs doesn't make sense for Drupal, but that good APIs tend to freeze themselves. That speaks to the heart of it, I think.
Since Drupal is open, there nothing preventing people from continuing to develop new modules for, say, 4.6 -- or even create a forked version and develop on that codebase.
We don't see much of that, I believe, because each new release brings such great new improvements to that core codebase. Like Ber points out, nobody is forced to upgrade. People choose to upgrade because they want that new performance, functionality, feature-set, etc. -- many of which would not even exist if Drupal had a frozen API or were dragging around redundant or outmoded code from years ago.
As a developer and consultant, I very much appreciate and understand the desire for a semi-frozen API. But I don't think Drupal is there yet, else we'd see forked versions building off of the branches, thriving on robust cores, being developed by people who don't see much improvement in the HEAD's API changes. I just figure upgrading is part of the deal, in order to benefit from the community's collective and various visions of what Drupal might yet be. I cut my teeth on Drupal with 4.5.1 and upgrading to 4.6. I muddled through it, despite knowing next to nothing at the time. It was hard and frustrating and had me in tears, but it beat the alternative.
To me, one of the appeals of Drupal is its ever-vital growth as a CMS (or whatever we want to call it). The day it stops that journey is the day I'll be looking for another solution. Convenience today is great, but potential for the future is where the real value lies for me.
May 17, 2006 - 18:17CivicSpace has spent a lot of money over the past two years porting a lot of code to new Drupal versions (probably more than any organization). There is no question in my mind that the sacrifice of backwards compatibility is well worth the new innovations it affords.
May 17, 2006 - 19:57I think that's an excellent, well-balanced analysis. If I may add my two cents:
The distinction between users who don't mind the lack of backward compatibility and those who insist upon it reminds me of the Crossing the Chasm phenomenon (in the books by Geoffrey A. Moore). This supports your conclusion that eventually Drupal will need to be more careful - as more users come from the right side of the chasm. But then, it's always possible there will be a Child of Drupal to assume the leading edge mantle.
As with Joe Murray, I'm a consultant new to Drupal. I was shocked at the 4.7 incompatibilities - not because they're there, but because it's a point release. The comments by merlinofchaos concerning version numbers are more or less a de facto standard of many years. I would, however, look at it from the other side. If you decide to make incompatible changes, then you also need to bump the version number (as opposed to picking the version number first, and allowing that decision to control the technical ones).
May 17, 2006 - 22:17I feel the need to correct one point.
I was shocked at the 4.7 incompatibilities - not because they're there, but because it's a point release.
This is completely incorrect. A check of the Drupal handbook and you will find this link about Drupal version numbers
Drupal version numbers or which version you should use
As you can tell from the page, version numbering is x.y.z
x.y indicates major release versioning so this makes 4.7.0 a major relase. This is clearly documented. Please let me know how we can make this clearer for new people.
May 19, 2006 - 05:13This is a tough topic. I've been using Drupal since 4.5 and I've been very frustrated by the upgrade path.
Sure, it's fantastic to stay at the bleeding edge of technology. And that clearly requires spending time developing that technology rather than fixing compatibility issues with older versions.
But that means that the longest-standing users are the ones hurt most by this practice. New users (or old users doing new installations on new sites) are thrilled to have the new features. But veteran users have a hell of a time upgrading their sites when modules that are key to their sites don't get updated with the core.
I'll spare you the details of my situation, but understand that this rapid release cycle is precisely what has pushed us away from Drupal for all future projects. The complexity of our planned ventures is sufficient that the development time (combined website and business development times) exceeds the Drupal release cycle.
We simply can't afford to lose 12 months of work every 12 to 24. Would we really lose all 12 months? No, but we can't predict what will break. And based on our past experience Drupal cares far more about new users than existing ones. The reality is that Drupal is always on the bleeding edge, but commercial implementations can not realistically support the rate of change without inordinately many developers of their own.
As a result, we're taking our "business" (development work) elsewhere.
- Jon
June 22, 2008 - 02:04Post new comment