Drupal
usaspending.gov using Drupal
Vivek Kundra, the CIO of the United States, unveiled the new IT spending dashboards at usaspending.gov earlier this week. Tim O'Reilly has all the details in his blog post titled Radical transparency: the new federal IT dashboard. In short, the dashboards are designed to help CIOs of individual government agencies get a handle on the effectiveness of government IT spending. The site was built with Drupal.
This is a fundamental change in the way government is going to be run, and it is great to see Drupal play a small role in that. Great stuff!
Acquia Search: benefits for site administrators
Yesterday we took the beta-wraps off of Acquia Search, and I followed up with a post about why Acquia Search matters for site visitors. We're still having some good discussions in the comments and the Twitter-sphere, but today I want to talk a bit more about the technical details. How does Acquia Search work, what does our infrastructure look like, and why is it a great deal for site owners?
Acquia Search is a hosted search service based on the Software as a Service (SaaS) model. The way it works is that Drupal sites push their content to the search servers hosted by Acquia. We index the content, build an index, and handle search queries. We provide the search results, facets, and content recommendations to your Drupal site over the network.
Your site's data is protected in transit by SSL and by HMAC authentication in the Acquia Network. Plain english? The data is encrypted so anyone snooping in the middle can't read it and the request is authenticated which means that the Acquia Network knows you sent the request you claimed to, and you know that messages received from the network are legitimate.
Acquia Search is built using the Open Source Lucene and Solr distributions from the Apache project. If you want to install, run and maintain Lucene and Solr yourself, and you have the resources to do so, you can. All the code, including our contributions to the Apache Solr integration modules for Drupal, are available as Open Source.
However, many organizations simply lack the Java expertise to deploy, manage and scale Java applications -- or their hosting environment may not accommodate it. Because Acquia Search is a hosted service, it takes away the burden of installation, configuration, and operational duties to keep the software fast, secure and up-to-date. That's our job.
As a reference, we've spent the last 9 months developing Acquia Search with the equivalent of three full-time employees. This also included setting up a billing system, integrating our support system, connecting it to the Acquia Network, performance testing and tuning, and more. Other Acquians helped out with the infrastructure, quality assurance, product management, design, and documentation. It was a non-trivial amount of work.
The result of these efforts is that we can launch any number of Solr farms on Amazon EC2. For high-performance and high-availability, each farm has a master Solr server and one or more slave Solr servers. A load balancer pushes content changes to the master Solr server, which are replicated by the slave servers. The load balancer makes sure that most regular search queries are done against the slave servers. Because multiple servers can handle your site's search requests, Acquia Search is fast and can scale, but it also means that Acquia Search is very robust because it can survive a server failure. As I wrote yesterday, Acquia Search is faster that Drupal's built-in search -- especially for large sites.
In most scenarios, several Drupal sites share a single Solr farm -- by sharing resources, we can offer a high-performance and high-availability search solution to small sites at relative cheap price point. For really big sites, we can provision a dedicated farm and scale out Solr so that it can handle millions of search queries.
Once you begin to use our search service you'll be able to disable Drupal's built-in core search. When you do this you reduce the amount of memory and processing power needed by your own infrastructure. As we've learned with big sites like drupal.org, Drupal's built-in search can bring a large site to its knees. With Acquia Search, you can avoid the drain.
On the front-end, we made significant contributions to the Apache Solr Search Integration modules on drupal.org. We helped add new features, improve the usability, and iron out a legion of bugs that cropped up during the beta period. The top-3 most active maintainers of the Apache Solr module are all Acquia employees, respectively Peter Wolanin, Robert Douglass, and Jacob Singh. As a result, Peter, Robert and Jacob are sometimes referred to as Acquia's three Apache Solr Musketeers.
Peter Wolanin (pwolanin), Robert Douglass and Jacob Singh work on Apache Solr integration as part of their job at Acquia. Peter and Jacob are part of the engineering team, but Robert can provide professional services related to Apache Solr.
All things combined, Acquia Search makes it staggeringly simple and low-cost to get better search on your site. You can get started in minutes and you don't have to worry about installing, upgrading, monitoring, or scaling the software. In short, we built an enterprise-quality, highly-available, secure, scalable, and fast indexing search solution that we believe Drupal was missing -- especially for the enterprise.
Acquia Search: benefits for visitors
Why will the visitors of your site care about Acquia Search? For a while now, I have Acquia Search installed on my personal site. To understand what Acquia Search can do for your site, have a look at what it has done for my site. While I have a very simple Drupal site, you should be able to experience some of the benefits of Acquia Search.
For example, search for "Drupal" on my site (use the search widget in the sidebar) and you can see the facets that allow you to filter the results by topic, location and industry. Using these facets, it should be pretty easy to find all the Fortune 500 Drupal sites that I blogged about in 2009, for example. Facets make search faster, making it very easy for your visitor to drill into results and to find what they are looking for.
Screenshot of Acquia Search's facet-based navigation as used on buytaert.net.
Acquia Search makes search easier because it is built on the principles of progressive disclosure. Instead of showing the visitor an initial page with lots of complicated options (see Drupal's advanced search options that almost no one uses), the facets are only shown after the initial search query. Plus, and this is really cool, facets are dynamically generated based on the search keywords. As such, they are relevant to what you're searching for.
Acquia Search provides a more powerful search because it is based on the renowned open source Lucene and Solr technologies from the Apache project. Not only do they sport better search algorithms, advanced content normalization, and a "did you mean?" feature, they also come with other great features such as word stemming, document search, range queries and more.
My favorite feature of Acquia Search, at least for use on this blog, is the "more like this" feature -- on node pages you can ask Acquia Search to suggest related content. I have been using it on my site for a while (see the block in the sidebar), and it has helped to keep visitors on my site longer. I occasionally find myself getting side-tracked by the "related links" -- it is a great way to re-discover old posts.
Screenshot of Acquia Search's content recommendations as used on buytaert.net.
Last but not least, our new service makes for better performance. We performed tests of searches on a Drupal site with over 10,000 nodes of content using a 3.2Ghz dual core server with 1.7 GB of RAM. With Acquia Search results were displayed in less than half a second, whereas the same results served from Drupal's built-in search took anywhere from 1.5 to 7.7 seconds. On the web, faster is better.
That makes for a lot of good reasons why the visitors of your site might care about Acquia Search. Tomorrow, I plan to write a more technical blog post about how Acquia Search works, how we made it that fast, and why it matters to site administrators (instead of site visitors). In the mean time, I recommend that you play around with the search feature on my site or that you sign up for a trial subscription. Have fun!
Acquia Search available commercially
It's a big day for us at Acquia. We finally took the beta-wraps off of Acquia Search, and made it available commercially as part of the Acquia Network. Thanks to the 250+ beta testers who helped make our hosted search service fit for use in production environments, including Brightcove, JackBe Developer Community, P-O-P Design, Wide Divots and others.
We used the beta period to look at the usage statistics, costs, and to talk to a lot of beta users to figure out the best pricing model for this service. We decided on the following:
Acquia Search is included for no additional cost in every Acquia Network subscription. Basic and Professional subscribers have one "search slice" and Enterprise subscribers have five "search slices". A slice includes the processing power to index your site, to do index updates, to store your index, and to process your site visitors' search queries. Each slice includes 10MB of indexing space - enough for a site with between 1,000 and 2,000 nodes. Customers who exceed the level included with their subscription may purchase additional slices. A ten-slice extension package costs an additional $1,000/year, and will cover an additional 10,000 - 20,000 nodes in an index of 100MB.
For my personal blog, which has about 900 nodes at the time of this writing, a Basic Acquia Network subscription ($349 USD/year) would give me all the benefits of Acquia Search, plus all the other Acquia Network services.
For some of you, this might sound like a lot of money, but we believe you get a lot of value in return. In my next couple of blog posts, I plan to outline the benefits of Acquia Search to your site visitors and to Drupal site administrators. Stay tuned!
Radio Netherlands Worldwide using Drupal
Radio Netherlands Worldwide (Radio Nederland Wereldomroep in Dutch or RNW for short) is a public radio and television network based in The Netherlands. Radio Netherlands Worldwide is a very old international public broadcaster, with regular transmissions that began in 1927 to the Dutch East Indies, now Indonesia.
What is interesting about the site is not the design or the implementation, but the fact that, after many years with Alterian (formerly Mediasurface), they switched to Drupal. Alterian is a supplier of proprietary content management systems, with their flagship product being Morello. The RNW started with the Internet early on (1992) and by doing so suffered from the law of the handicap of a head start: a history of dated, proprietary CMS-es that held them back from moving to the more current software.
RNW selected Drupal because of its multi-lingual capabilities (they support up to 6 languages, including Chinese and Arabic content) and Drupal's flexibility and agility. The migration to Drupal was done by Dutch Open Projects and took 3 months with a team of 5 people.
Bert Boerland, project manager at Dutch Open Projects, wrote the following on the switch to Drupal: By itself, the fact that Radio Netherlands Worldwide switches from a proprietary CMS towards an Open Source CMS is not the biggest news. However, the switch is a milestone since it symbolizes that companies that didn't look to Open Source and only listened to the proprietary "prietpraat" are moving over! [The word prietpraat is Dutch and translates roughly to "childish non-sense".]
With both RNW and the Dutch public broadcaster NCRV using Drupal, and through Acquia's partnership with Woodwing (Woodwing is a Dutch company), the media landscape in Europe's low-lands has some critical mass to push Drupal into more traditional broadcasting companies. Drupal starts to disrupt the traditional, proprietary web content management space increasingly more.
With an editorial staff of over one hundred people, RNW publicizes dozens of postings a day including their own video and audio, and will soon incorporate even more, including user-generated content and content from their five-thousand-plus partner stations around the world. To keep all this content on track, RNW choose Drupal, coupled with the usual contributed modules (CCK and Views, FCKeditor, Pathauto, etc). They use Organic Groups as their core for separating and integrating their eleven editorial staffs. RNW gets lots of traffic from around the world and so, gets lots of spam, they are also a paying Mollom user; Mollom blocks hundreds of spam comments a day for them.
First Belgian DrupalCamp
Next weekend, on Saturday June 27, the French-speaking Drupal community in Belgium is organizing Belgium's very first DrupalCamp. The event takes place in Louvain-la-Neuve which makes it pretty accessible for both Dutch and French-speaking Belgians. Details are available on http://drupalfr.be. Due to the work of Gilles Bailleux, the event got some great press in this weekend's edition of the Belgian newspaper, La Libre Belgique. Both a screenshot and a PDF are provided below.
© La Libre Belgique. Download PDF version.
It's always great to meet Drupal people, so I currently plan to go to Utrecht on Friday, to Louvain-la-Neuve on Saturday, and back to Utrecht on Sunday to help with the Drupal 7 usability code sprint. It's just great to see so many community events being organized -- we've gotten to a point where there are often Drupal events happening at the same time in different places in the world.
Drupal 7 usability code sprint in The Netherlands
On Friday, there is a Drupal meetup happening in Utrecht, The Netherlands. I'm planning to attend so looking forward to meet some of you there.
On Saturday and Sunday, a smaller group of core developers will meet in the offices of One Shoe in downtown Utrecht to work on the ongoing Drupal 7 usability efforts. According to the event page on groups.drupal.org, confirmed attendees for the sprint include Leisa Reichelt, Mark Boulton, Gábor Hojtsy (goba), Damien Tournoud (DamZ), Erik Stielstra (sutharsan), Bojhan Somers (bojhan), Roy Scholten (yoroy), Bart Feenstra (Xano), Gaele Strootman (gaele), Kristjan Jansen (kika), Thomas Moseler (eigentor), Konstantin Kaefer (kkaefer), Philip Vergunst (skilip), Willem Mol (Whatdoesitwant), Berend de Boer (berend), Maarten Verbaarschot (mverbaar), Johannes Haseitl (derhasi), Steve De Jonghe (seutje), Clemens Tolboom (clemens.tolboom), Thijs Zoon, and Floris Derksen. I plan to stop by on Sunday as well.
Though the open sign-up for the code sprint on Saturday and Sunday has already closed due to the number of attendees already confirmed, contact Thomas Moseler (eigentor) if you want to attend and he may be able to squeeze you in. He's also actively looking for sponsors to help some European core developers to attend the Utrecht activities; if you can help, please send him an e-mail. Thanks!
