Acquia Search
Acquia Search versus Drupal search
It's been several days since we launched Acquia Search commercially. After reviewing the press, articles, comments, and tweets, I wanted to address the question of why we seem to care so much about search and why we can't simply improve Drupal's built-in search module. These questions came up during the beta test period as well, and have even resonated with the WordPress community on Matt Mullenweg's blog. I feel they are important questions to address.
I've already partially answered these questions in two recent blog posts -- why Acquia Search matters for site administrators and why Acquia Search matters for site visitors -- but there is more to it.
First, at the end of the day, search is a hard but important problem. This is reflected by the size of the search market. Some have estimated the search market to be at least as big as the web content management market. The leading providers of site search technology such as Autonomy, FAST and Endeca have built large, successful businesses supplying search technology to the enterprise. Last year, FAST was acquired by Microsoft for $1.2 billion. Gartner forecasts that the enterprise search market will grow to more than $1.1 billion in total software revenue by 2011 (excluding professional service revenues). For many people in the Drupal community, these data points will probably come as a surprise.
Reality is that for a certain class of websites -- like intranets or e-commerce websites -- search can be the most important feature of the entire site. Faceted search can really increase your conversions if you have an e-commerce website, or can really boost the productivity of your employees if you have a large intranet. For those organizations, Drupal's built in search is simply not adequate. We invested in search because we believe that for many of these sites, enterprise-grade search is a requirement.
Secondly, why don't we just implement improvements in Drupal's core search module? As I've noted, search is a difficult problem -- it is hard for Drupal to compete with enterprise-grade search engines, to keep up with advances in search technology, and to do both while continuing to run in shared hosting environments. Instead, Acquia Search leverages the Open Source Lucene and Solr distributions from the Apache project.
The search module shipped with Drupal core has its purpose and target audience. It isn't right for everyone, just as Acquia Search is not for everyone. Both are important, not just for the Drupal community at large, but also for many of Acquia's own customers. Regardless, there is no question that we need to keep investing and improving Drupal's built-in search. The search module that is built into Drupal 7 already has improvements over the one in Drupal 6, in part because of Acquia's support of the Search sprint in Minnesota.
I'm hopeful that we can scale up our investments in Acquia Search as we grow the search component of our business. There is a lot more we can do, so I'd like to see us become active contributors to Apache Lucene and Solr, as well as continue to ramp our contributions to the different Apache Solr projects on drupal.org, as well as Drupal core's built-in search.
Acquia Search: benefits for site administrators
Yesterday we took the beta-wraps off of Acquia Search, and I followed up with a post about why Acquia Search matters for site visitors. We're still having some good discussions in the comments and the Twitter-sphere, but today I want to talk a bit more about the technical details. How does Acquia Search work, what does our infrastructure look like, and why is it a great deal for site owners?
Acquia Search is a hosted search service based on the Software as a Service (SaaS) model. The way it works is that Drupal sites push their content to the search servers hosted by Acquia. We index the content, build an index, and handle search queries. We provide the search results, facets, and content recommendations to your Drupal site over the network.
Your site's data is protected in transit by SSL and by HMAC authentication in the Acquia Network. Plain english? The data is encrypted so anyone snooping in the middle can't read it and the request is authenticated which means that the Acquia Network knows you sent the request you claimed to, and you know that messages received from the network are legitimate.
Acquia Search is built using the Open Source Lucene and Solr distributions from the Apache project. If you want to install, run and maintain Lucene and Solr yourself, and you have the resources to do so, you can. All the code, including our contributions to the Apache Solr integration modules for Drupal, are available as Open Source.
However, many organizations simply lack the Java expertise to deploy, manage and scale Java applications -- or their hosting environment may not accommodate it. Because Acquia Search is a hosted service, it takes away the burden of installation, configuration, and operational duties to keep the software fast, secure and up-to-date. That's our job.
As a reference, we've spent the last 9 months developing Acquia Search with the equivalent of three full-time employees. This also included setting up a billing system, integrating our support system, connecting it to the Acquia Network, performance testing and tuning, and more. Other Acquians helped out with the infrastructure, quality assurance, product management, design, and documentation. It was a non-trivial amount of work.
The result of these efforts is that we can launch any number of Solr farms on Amazon EC2. For high-performance and high-availability, each farm has a master Solr server and one or more slave Solr servers. A load balancer pushes content changes to the master Solr server, which are replicated by the slave servers. The load balancer makes sure that most regular search queries are done against the slave servers. Because multiple servers can handle your site's search requests, Acquia Search is fast and can scale, but it also means that Acquia Search is very robust because it can survive a server failure. As I wrote yesterday, Acquia Search is faster that Drupal's built-in search -- especially for large sites.
In most scenarios, several Drupal sites share a single Solr farm -- by sharing resources, we can offer a high-performance and high-availability search solution to small sites at relative cheap price point. For really big sites, we can provision a dedicated farm and scale out Solr so that it can handle millions of search queries.
Once you begin to use our search service you'll be able to disable Drupal's built-in core search. When you do this you reduce the amount of memory and processing power needed by your own infrastructure. As we've learned with big sites like drupal.org, Drupal's built-in search can bring a large site to its knees. With Acquia Search, you can avoid the drain.
On the front-end, we made significant contributions to the Apache Solr Search Integration modules on drupal.org. We helped add new features, improve the usability, and iron out a legion of bugs that cropped up during the beta period. The top-3 most active maintainers of the Apache Solr module are all Acquia employees, respectively Peter Wolanin, Robert Douglass, and Jacob Singh. As a result, Peter, Robert and Jacob are sometimes referred to as Acquia's three Apache Solr Musketeers.
Peter Wolanin (pwolanin), Robert Douglass and Jacob Singh work on Apache Solr integration as part of their job at Acquia. Peter and Jacob are part of the engineering team, but Robert can provide professional services related to Apache Solr.
All things combined, Acquia Search makes it staggeringly simple and low-cost to get better search on your site. You can get started in minutes and you don't have to worry about installing, upgrading, monitoring, or scaling the software. In short, we built an enterprise-quality, highly-available, secure, scalable, and fast indexing search solution that we believe Drupal was missing -- especially for the enterprise.
Acquia Search: benefits for visitors
Why will the visitors of your site care about Acquia Search? For a while now, I have Acquia Search installed on my personal site. To understand what Acquia Search can do for your site, have a look at what it has done for my site. While I have a very simple Drupal site, you should be able to experience some of the benefits of Acquia Search.
For example, search for "Drupal" on my site (use the search widget in the sidebar) and you can see the facets that allow you to filter the results by topic, location and industry. Using these facets, it should be pretty easy to find all the Fortune 500 Drupal sites that I blogged about in 2009, for example. Facets make search faster, making it very easy for your visitor to drill into results and to find what they are looking for.
Screenshot of Acquia Search's facet-based navigation as used on buytaert.net.
Acquia Search makes search easier because it is built on the principles of progressive disclosure. Instead of showing the visitor an initial page with lots of complicated options (see Drupal's advanced search options that almost no one uses), the facets are only shown after the initial search query. Plus, and this is really cool, facets are dynamically generated based on the search keywords. As such, they are relevant to what you're searching for.
Acquia Search provides a more powerful search because it is based on the renowned open source Lucene and Solr technologies from the Apache project. Not only do they sport better search algorithms, advanced content normalization, and a "did you mean?" feature, they also come with other great features such as word stemming, document search, range queries and more.
My favorite feature of Acquia Search, at least for use on this blog, is the "more like this" feature -- on node pages you can ask Acquia Search to suggest related content. I have been using it on my site for a while (see the block in the sidebar), and it has helped to keep visitors on my site longer. I occasionally find myself getting side-tracked by the "related links" -- it is a great way to re-discover old posts.
Screenshot of Acquia Search's content recommendations as used on buytaert.net.
Last but not least, our new service makes for better performance. We performed tests of searches on a Drupal site with over 10,000 nodes of content using a 3.2Ghz dual core server with 1.7 GB of RAM. With Acquia Search results were displayed in less than half a second, whereas the same results served from Drupal's built-in search took anywhere from 1.5 to 7.7 seconds. On the web, faster is better.
That makes for a lot of good reasons why the visitors of your site might care about Acquia Search. Tomorrow, I plan to write a more technical blog post about how Acquia Search works, how we made it that fast, and why it matters to site administrators (instead of site visitors). In the mean time, I recommend that you play around with the search feature on my site or that you sign up for a trial subscription. Have fun!
Acquia Search available commercially
It's a big day for us at Acquia. We finally took the beta-wraps off of Acquia Search, and made it available commercially as part of the Acquia Network. Thanks to the 250+ beta testers who helped make our hosted search service fit for use in production environments, including Brightcove, JackBe Developer Community, P-O-P Design, Wide Divots and others.
We used the beta period to look at the usage statistics, costs, and to talk to a lot of beta users to figure out the best pricing model for this service. We decided on the following:
Acquia Search is included for no additional cost in every Acquia Network subscription. Basic and Professional subscribers have one "search slice" and Enterprise subscribers have five "search slices". A slice includes the processing power to index your site, to do index updates, to store your index, and to process your site visitors' search queries. Each slice includes 10MB of indexing space - enough for a site with between 1,000 and 2,000 nodes. Customers who exceed the level included with their subscription may purchase additional slices. A ten-slice extension package costs an additional $1,000/year, and will cover an additional 10,000 - 20,000 nodes in an index of 100MB.
For my personal blog, which has about 900 nodes at the time of this writing, a Basic Acquia Network subscription ($349 USD/year) would give me all the benefits of Acquia Search, plus all the other Acquia Network services.
For some of you, this might sound like a lot of money, but we believe you get a lot of value in return. In my next couple of blog posts, I plan to outline the benefits of Acquia Search to your site visitors and to Drupal site administrators. Stay tuned!
Acquia Search screencast with RedMonk
In this screencast, Bryan House from Acquia discusses Acquia Search with Michael Coté from RedMonk. Great demo, Bryan!
Acquia Search goes public beta
At Acquia, we've launched Acquia Search in public beta today.
As I've previously announced, Acquia Search is a hosted Acquia Network service, delivering the capabilities of the Apache Solr as a web service to Drupal websites of any size.
Acquia Search is 100% free for now, and here is our marketing pitch: "Acquia Search can be installed as a module on any Drupal 6 site, and enhances a site's search experience with faceted search navigation, content recommendations, and configurable results weighting, all delivered through a redundant hosted service infrastructure.". Put a bit more directly, Acquia Search is simply a great way to supercharge your site's search experience, without having to invest in your own Solr engine and development.
Though we're not sure how long our beta program will last, one of our motivations in providing a free beta is to help figure out the pricing model. There will likely be two main components to our price structure -- the number of nodes on a site and the number of search queries. The free beta will allow us to get a better feel for that.
But what about Drupal's built-in search module? This announcement doesn't mean that we'll stop enhancing it -- in fact, the Drupal 7 search module already has some improvements over the one in Drupal 6. There's lots of room in the "search" arena, with lots of different use cases, and it is hard for the core search module to compete with commercial-grade search engines while running on shared hosting environments. For a certain class of sites, including drupal.org, Drupal's built-in search is simply not an option, and we think Acquia Search can be a great fit for those sites.
Acquia Search is only possible because of the fantastic work of the many people who have helped make it a reality. The Apache Solr Search Integration project on Drupal.org, for instance, reflects the work of many community members, including Robert Douglass, Peter Wolanin, Jacob Singh, Damien Tournoud, and more. Thanks!
Acquia 2009 roadmap
Now that the Acquia Network and Acquia Drupal are available, it is a good time to provide more visibility into where Acquia is headed. Just like we did early on in Acquia's life when we announced "Carbon" (now called "Acquia Drupal") and "Spokes" (now called the "Acquia Network"), we'd like to provide some transparency and visibility in our product strategy for 2009. This post provides a window into those plans.
As we started making plans for 2009, we realized that the best thing we could do, not just for us but also for our partners and the Drupal community at large, is to help tear down the barriers to using Drupal so that it can fulfill its full potential. As a result, much of what Acquia will do in 2009 relates to improving Drupal's adoption rate (e.g. improved usability, increasing awareness, better scalability, commercial-grade support, etc).
Here are Acquia's biggest projects for the year ... just like we did in 2008 with "Carbon" and "Spokes", we're using code names for most of our new projects.
Acquia Front Lawn: Drupal 7 usability
With the success of the drupal.org redesign in mind, I wondered if it would be possible for Mark Boulton and Leisa Reichelt to help us improve usability in Drupal itself. Since one of Acquia's key goals is to help expand Drupal adoption, and improving Drupal's usability is key to that, we decided to hire Mark Boulton to help with Drupal 7. We've asked Mark and Leisa to do all their work out in the open, in the Drupal community, so that the community can be involved and give input every step of the way. They have already been brainstorming with some members of Drupal's usability team, but the actual work will start at DrupalCon DC in two weeks.
Acquia Fields: scalable Drupal infrastructure
As a Drupal site gets more popular and bigger, keeping the site running and making it scale can be a huge challenge. Drupal can scale well, but expertise required to make Drupal scale well can be expensive and hard to find. We will address this problem with Acquia's custom Drupal hosting -- code-named Acquia Fields.
We built acquia.com in the cloud (using Amazon EC2), and have learned what works in the cloud and what doesn't. And we've had a rapidly growing number of requests to help customers build out similar infrastructure for their large-scale sites. We'll continue to build this incrementally, but our long term vision includes virtually unlimited scalability, automatic off-site backup and recovery, continuous site performance monitoring, and more. All of this will be backed by Acquia's support team.
Acquia Gardens: Drupal for everyone
Many individuals and organizations want a killer web site, but have no idea that Drupal is a great way to build one or to connect with other websites. Even if they did hear about Drupal, few non-technical people succeed in installing and hosting a Drupal site. In much the same way that Wordpress.com and Ning make it easy for people to start blogging or set up a social network, Acquia Gardens will provide an on-ramp for people to experience the awesome power of a Drupal based social publishing website. Our goal is to make the base service free of charge, and to introduce Drupal to millions of new users. Having a free entry point is essential to promoting viral adoption via word of mouth, which will help dramatically increase awareness of what Drupal has to offer. To make the services sustainable, we will charge for premium services and features.
A fundamental tenet of our strategy is the freedom to grow, migrate, or leave. We believe that site owners should be able to start with Acquia Gardens and migrate their site to Acquia Fields if they need more features, or to their own hosting provider if they prefer. To make that possible, you'll be able to export your Acquia Gardens site.
Acquia Network at the center
If we succeed in building the products above, we will have 3 deployment models for Drupal: (i) using Drupal, or Acquia Drupal, on your own infrastructure, (ii) using Drupal on Acquia Fields, and (iii) as part of Acquia Gardens. At the center of these 3 deployment options lives the Acquia Network. The Acquia Network provides three types of services for site owners: technical support, site management services (site traffic and usage statistics, uptime monitoring), and services to add valuable functionality to your site (powerful content searching, spam blocking, media delivery). The need for these services are universal regardless of how you use Drupal or where Drupal is hosted.
Acquia Search
What we offer today in the Acquia Network is just a start. We have a long list of ideas for new services that will enhance Acquia Drupal sites and generally make life easier for site owners. The newest member of the Acquia Network services will we our hosted search offering based on Apache Solr. Acquia Search will provide Drupal site owners multi-site search, better results, faster performance, document search, federated search, and more. We already blogged about this, and we are recruiting beta testers. We think it has big value for Drupal users.
Conclusion
There are other great things that we are working on, but I've decided to focus on the broad initiatives that we're undertaking in 2009.
As always, much of the work we'll do, we'll do in the community, and we hope that others will join us as partners in realizing these plans, because we need the collaboration and support of many people outside the company to make it all work well. If you aspire to some of the same business objectives, we're open to joining forces around common goals.
There is a lot of work ahead to implement this strategy. Needless to say, it will be a multi-year effort to implement everything as envisioned. The game may change along the way, however, the groundwork is in place and we expect to be in the market with incremental progress as soon as possible. In the mean time, we'd love to get your feedback on what we have planned so we can make adjustments and improvements.
We can't wait to tell you more -- at the right time. In the mean time, you can find our roadmap at http://acquia.com/community/projects/acquia-2009-roadmap.
