I used XDebug to profile the behavior of Drupal, and to study the interaction with the database server. I aggregated the profile information of 100 requests to the main page using the "Apache, mod_php, PHP4, APC" configuration used for previous benchmark experiments. More information about my experimental setup is available at that page. XDebug generates a trace file with all the profile information which I visualized using KCacheGrind.
Drupal has a page cache mechanism which stores dynamically generated web pages in the database. By caching a web page, Drupal does not have to create the page each time it is requested. Only pages requested by anonymous visitors (users that have not logged on) are cached. Once users have logged on, caching is disabled for them since the pages are personalized in various ways. Because this represents two different modes of operation, I investigated Drupal's behavior with and without page caching.
With page caching
This figure shows how much time is spent in each function when page caching is enabled. The functions are sorted by the 'Self' column, which shows the time spent in each function without the time spent in its children. The second column shows how often the function was called to serve 100 requests to the main page.
We observe that more than 45% (14.37% + 14.18% + 8.9% + 5.54% + 2.9% + ...) of the execution time is spent in the database related functions. Sending the SQL queries to the database server and waiting for the results takes 14% of the total execution time. Drupal preparing the queries (eg. database prefixing, sanitizing the input to prevent SQL query injection, etc) takes more than 31% of the total execution time. We should look into optimizing the functions that prepare the queries (
The figure above depicts that PHP's
mysql_query() function is called 1401 times. This means that we need 14 SQL queries to serve a cached page. This is where these SQL queries come from:
This figure shows the Drupal functions responsible for querying the database when serving 100 cached pages. The first column shows how much time is spent in the calls to <code>db_query()</code>. The second column shows how many times each function queried the database and the last column shows the functions' names and source files.
Without page caching
This figure shows how much time is spent in each function when page caching is disabled. The functions are sorted by the 'Self' column, which shows the time spent in each function without the time spent in its children. The second column shows how often the function was called to serve 100 requests to the main page.
This figure shows the Drupal functions responsible for querying the database when serving 100 pages. The first column shows how much time is spent in the calls to <code>db_query()</code>. The second column shows how many times each function queried the database and the last column shows the functions' names and source files.
About six years ago I started working on Drupal. Drupal, at that time, was an experimental platform that helped me explore new web technologies from my student dorm. Contrast this with the present. Today, there are hundreds of people contributing to the project, building and relying on that foundation, and hundreds of thousands of people downloading it. What started as a hobby project is now starting to get on the radar of some of the bigger projects and players ... It is no longer the casual hobby project it used to be.
It is fair to say that Drupal's growth makes for some interesting questions, both for me personally, and for the Drupal community at large. It makes me feel increasingly responsible, and that certainly adds some pressure. How to help run this thing as it continues to grow? Do we need a Drupal Foundation or not? How should I deal with my growing sense of responsibility? Or how to deal with being labeled an anti-Bill Gates?
I'm particularly interested to hear what other projects and people in a similar position do, or have done. So in an effort to make some connections and relationships with leaders in the FOSS community, I'll be taking a "Drupal road trip" to the San Francisco Bay Area from June 25th to June 30th.
Jeff Robbins from Lullabot is helping me contact people and set up my schedule. We set up personal meetings with some of the smartest people in the FOSS and internet community:
- Tim O'Reilly (Founder and CEO of O'Reilly & Associates)
- Chris DiBona (Open Source Programs Manager at Google)
- Mitch Kapor (Co-founder of Lotus-1-2-3, founder of the Open Source Applications Foundation, co-founder of the Electronic Frontier Foundation, chair of the Mozilla Foundation)
- Jeffrey Veen (Google / Measure Map / Adaptive Path)
- Channel Wheeler and Bradley Greenwood (Yahoo!)
- Janice Fraser (CEO of Adaptive Path)
- Guido van Rossum (Founder of the Python project, Google)
- Larry M. Augustin (CEO of VA Linux)
- Anders Tjernlund (VP of Support Services at SpikeSource)
- Brian Behlendorf (co-founder of the Apache Foundation)
If you would like to connect us with others in the Bay Area that should be on this list, please contact us.
These meetings will give me a chance to talk to these people about what is happening with Drupal and the Drupal community, and get a chance to promote all the great work we've done together. We're going to try and make sure that Drupal can be a little more connected with the larger FOSS community. Maybe it will open up further possibilities for collaboration and support. As always, we'll let things go naturally. Most importantly, I hope to learn from these people and that alone is going to be an invaluable experience.
As well as all of these "official" meetings, I would love to connect with the local Drupal community. We're leaving at least Thursday evening (the 29th) open for a Drupal meetup, but there should be other opportunities to hang out. Check the Bay Area Drupal group for details on time and place.
André Gilain interviewed me about Drupal for Trends, a Belgian business magazine. You can find the article in the June 15 issue of Trends Tendances (French version), or you can read it in full by clicking the link below.
<a href="/files/trends06-interview.pdf">PDF version</a>, © Trends.
This is really good publicity for Drupal, and it is exciting to see our work getting so much recognition.
I'm not sure that I like being called the "anti-Bill Gates" -- it is not like I'm a modern hippy fighting windmills, am I? I wish that the article was more about Drupal and the Drupal community, and less about me. Credit should be given where credit is due: Drupal's successes should be attributed to the Drupal community, of which I am just one part.