Dries Buytaert

Acquia product strategy and vision

In my Acquia 2010 retrospective, I promised to write a bit more about Acquia's product strategy. This blog post provides a high level view of the vision that we've been working towards for the last 3 years, and explains how Acquia can help simplify your web strategy.

The web: it's currently a mess

Ten years ago, the average organization had one website. Since then, doing business through the web has become more complex and have introduced a diverse set of needs. If you're like most organizations the number of sites you have is large and continues to grow at a rapid clip.

For most organizations, one tool could not historically get the job done, so they kept multiple tools in their toolbox – whether they intended to or not. The situation can be quite a mess, and is unfortunately a common scenario in many enterprises.

Each site has unique needs

Most of these sites are vastly different in terms of scale, functionality, complexity and longevity. Some sites are under continuous development while other sites are only around for a couple of weeks or months. Some of the websites are owned by the company's IT department and hosted internally, while other websites may be owned by their marketing department and hosted externally. As a result, the level of investment and the time to market requirements are usually very different.

Standardize on Drupal to save costs

CIOs – facing cost-cutting pressures and the need to streamline their resources – are now addressing the reality of running twenty different content management systems on twenty different stack configurations as an expensive, unnecessary burden for the organization. They have always known that there were cost savings to be made if they standardize on a single platform, but have never felt the confidence in a single platform to suit all of their needs across their organization.

Drupal has the required features to accomplish this today. This is more than a vision – it is reality. Every day, more organizations are standardizing on Drupal.

By standardizing on Drupal, organizations can reduce training costs, reduce maintenance costs, streamline security, and optimize internal resources – all without sacrificing quality or requirements. Standardizing on Drupal certainly doesn't mean every single system needs to be Drupal. Even going from 20 different systems to 10 or to 5 different systems still translates to dramatic cost savings. It goes without saying that you need to be smart about what makes sense to standardize on Drupal, and what not to standardize on Drupal. With our vast community of contributors, Drupal continues to become better and better and the feasibility for an organization to standardize on Drupal continues to improve over time.

Drupal distributions help adoption

Drupal distributions are an important part of helping organizations adopt Drupal. Drupal distributions are complete, ready-to-use solutions built on Drupal. Just install and go.

Drupal Commons is a Drupal distribution for social business software; it provides organizations a complete solution for forming collaborative communities. Similarly, Open Publish is a Drupal distribution optimized for news publishing. Acquia sees expansion of distributions as critically important to the future growth of Drupal. With that, we are acting as a software publishers for these and other distributions developed by partners within the Drupal community; supporting the marketing, promotion, support, and ongoing development of distributions to extend the capability of the companies who have incubated these incredible products.

Add the Acquia Network for support and cloud services

To help organizations adopt and standardize on Drupal, we created the Acquia Network to provide a suite of Drupal support, knowledge, and web development and maintenance tools to help build, manage and extend Drupal websites.

The Acquia Network is your connection to a team of Drupal experts, available 24x7, and backed by Acquia's engineering and professional services team. As an Acquia Network subscriber, you can submit help tickets, search our knowledge base and contribute in our subscriber forums.

The Acquia Network also provides you access to a number of cloud-based services. Services like heartbeat monitoring, software update management, and soon to be released integration with New Relic provide visibility into your site's performance and help with site management. Other services, like Acquia Search and Mollom, extend the functional capabilities of your sites.

We are in the middle of a massive redesign of the Acquia Network and many of the services you use through the Acquia Network today (including the Acquia Library, a broad collection of tips, tricks, how-to's, and resources for Drupal developers and site owners). Through the Acquia Network you will soon have the ability to easily access a growing list of third-party services, with many available at no additional charge. We already offer many third-party services (e.g. Mollom for spam filtering, New Relic for application profiling, etc), but we'll soon be opening up the Acquia Network as a ‘service delivery platform' and marketplace for additional services. In the works for release over the next few months are mobile design tools from Mobify, analytics, video services, marketing tools, and more.

Interested in adding your service to the Acquia Network? In the future, we will roll out APIs and infrastructure (e.g. billing) to enable other organizations to deliver their cloud-services to any Drupal site through the Acquia Network.

Add Acquia Hosting, a Drupal Platform-as-a-Service

For large websites that require custom code, high availability, on-demand elasticity or release management tools (i.e. staging and production workflows), we recommend Acquia Hosting, our Drupal-platform-as-a-service (Drupal PaaS).

Acquia Hosting is an extension of the Acquia Network, so if you need help scaling your site or debugging a problem, Acquia Client Advisors are always available to help. Through the Acquia Network, we also provide a number of Acquia Hosting specific e-services, including backups, database rollbacks, staging environments, version control for code management, and more.

Going forward you can expect even more developer tools and self-service tools to be added to Acquia Hosting, as well as more critical features for large scale sites, including improved security and code workflow options.

Add Drupal Gardens for rapid micro-site development

All sites are different. Not all your organization's website need the scale, functionality, complexity or longevity of your most important websites. A lot of times you have smaller sites that you may want to roll-out quickly, preferably without having to involve IT.

For that, we built Drupal Gardens, a Drupal-as-a-service platform that makes building Drupal websites as simple as point and click. Built on Drupal 7, Drupal Gardens brings the freedom and innovation you expect from open source without having to worry about installing, hosting or upgrading your Drupal site.

Our mission for Drupal Gardens is to allow site builders to go from design to online in minutes instead of days or weeks. To help, we provide an ever-growing library of site templates and themes to start from. We believe it will be the best platform for your smaller sites that complement your primary web properties.

For organizations that need to manage tens, if not hundreds, of small websites, we're building ‘Enterprise Drupal Gardens'. It provides site provisioning, site management, single sign-on, multi-site dashboards and organization wide templates and themes to maintain consistent branding.

Host your own sites, if you prefer

One of the biggest advantages of using Open Source software is that there are no limits to how you use the software. Some organizations prefer to host some of their own sites. The Acquia Network is able to plug in into your site, regardless of where it is hosted.

No lock in with "Open SaaS"

Almost all Software as a Service (SaaS) providers employ a proprietary model – they might allow you to export your data, but they usually don't allow you to export the underlying code. Users of Drupal Gardens are able to export their Drupal Gardens site – the code, the theme and data – and move of the platform to any Drupal hosting environment. By doing so, we provide people an easy on-ramp but we allow them to grow beyond the capabilities of Drupal Gardens without locking them in.

We call this "Open SaaS" or Software as a Service done right based on Open Source principles – it offers a much more secure and low-cost alternative to proprietary counterparts.

Conclusions

I've highlighted some of our key products and services in this blog post and will bring you a more detailed white paper focusing on Acquia's vision. Stay tuned!

Acquia Search: an update after one year

About 20 months ago, at Acquia, we began working on a hosted offering for Apache Solr, an open source enterprise search platform from the Apache Lucene project. Exactly one year ago, we launched it commercially as Acquia Search. Time and the public reaction have proven that we made the right choice. In the past year, Apache Solr has received a tremendous amount of traction in the Drupal community. Most large sites launched recently use Apache Solr because it provides a faster, more scalable search solution, as well as improved search accuracy and more features than the built-in search features of Drupal's core.

If you want to install, run and maintain Apache Solr yourself -- assuming you have the resources required -- you can of course do so. However, many organizations lack the technical expertise to deploy, maintain and scale Java applications. Even if they do have the resources, it's often cheaper to use Acquia Search. Acquia Search has been part of our overall plan to sell simplicity and enhance the experience of using Drupal. Today, the majority of our customers that subscribe to the Acquia Network, which includes very large Drupal sites, actively use Acquia Search instead of maintaining their own or using Drupal core's built-in search. In the past three months we have handled about 20 million search requests on behalf of our customers. These are important proof-points of our strategy.

The growth in popularity of Apache Solr and the story of Acquia Search haven't finished, though. This week we released some excellent new features for Acquia Search which we believe will further help drive adoption of Apache Solr and Acquia Search. We added support for attachment indexing (e.g. search PDF and Word documents), multi-site search (i.e. search multiple Drupal sites at once), and other additions. For more details on this latest release of Acquia Search, check out Peter Wolanin's blog post on the subject. I think our customers will be quite pleased at the improvements we've made in this release. And if you're not using Apache Solr or Acquia Search, you should seriously consider implementing it. It's cool stuff. :-)

Acquia Search versus Drupal search

It's been several days since we launched Acquia Search commercially. After reviewing the press, articles, comments, and tweets, I wanted to address the question of why we seem to care so much about search and why we can't simply improve Drupal's built-in search module. These questions came up during the beta test period as well, and have even resonated with the WordPress community on Matt Mullenweg's blog. I feel they are important questions to address.

I've already partially answered these questions in two recent blog posts -- why Acquia Search matters for site administrators and why Acquia Search matters for site visitors -- but there is more to it.

First, at the end of the day, search is a hard but important problem. This is reflected by the size of the search market. Some have estimated the search market to be at least as big as the web content management market. The leading providers of site search technology such as Autonomy, FAST and Endeca have built large, successful businesses supplying search technology to the enterprise. Last year, FAST was acquired by Microsoft for $1.2 billion. Gartner forecasts that the enterprise search market will grow to more than $1.1 billion in total software revenue by 2011 (excluding professional service revenues). For many people in the Drupal community, these data points will probably come as a surprise.

Reality is that for a certain class of websites -- like intranets or e-commerce websites -- search can be the most important feature of the entire site. Faceted search can really increase your conversions if you have an e-commerce website, or can really boost the productivity of your employees if you have a large intranet. For those organizations, Drupal's built in search is simply not adequate. We invested in search because we believe that for many of these sites, enterprise-grade search is a requirement.

Secondly, why don't we just implement improvements in Drupal's core search module? As I've noted, search is a difficult problem -- it is hard for Drupal to compete with enterprise-grade search engines, to keep up with advances in search technology, and to do both while continuing to run in shared hosting environments. Instead, Acquia Search leverages the Open Source Lucene and Solr distributions from the Apache project.

The search module shipped with Drupal core has its purpose and target audience. It isn't right for everyone, just as Acquia Search is not for everyone. Both are important, not just for the Drupal community at large, but also for many of Acquia's own customers. Regardless, there is no question that we need to keep investing and improving Drupal's built-in search. The search module that is built into Drupal 7 already has improvements over the one in Drupal 6, in part because of Acquia's support of the Search sprint in Minnesota.

I'm hopeful that we can scale up our investments in Acquia Search as we grow the search component of our business. There is a lot more we can do, so I'd like to see us become active contributors to Apache Lucene and Solr, as well as continue to ramp our contributions to the different Apache Solr projects on drupal.org, as well as Drupal core's built-in search.

Acquia Search: benefits for site administrators

Yesterday we took the beta-wraps off of Acquia Search, and I followed up with a post about why Acquia Search matters for site visitors. We're still having some good discussions in the comments and the Twitter-sphere, but today I want to talk a bit more about the technical details. How does Acquia Search work, what does our infrastructure look like, and why is it a great deal for site owners?

Acquia Search is a hosted search service based on the Software as a Service (SaaS) model. The way it works is that Drupal sites push their content to the search servers hosted by Acquia. We index the content, build an index, and handle search queries. We provide the search results, facets, and content recommendations to your Drupal site over the network.

Your site's data is protected in transit by SSL and by HMAC authentication in the Acquia Network. Plain english? The data is encrypted so anyone snooping in the middle can't read it and the request is authenticated which means that the Acquia Network knows you sent the request you claimed to, and you know that messages received from the network are legitimate.

Acquia Search is built using the Open Source Lucene and Solr distributions from the Apache project. If you want to install, run and maintain Lucene and Solr yourself, and you have the resources to do so, you can. All the code, including our contributions to the Apache Solr integration modules for Drupal, are available as Open Source.

However, many organizations simply lack the Java expertise to deploy, manage and scale Java applications -- or their hosting environment may not accommodate it. Because Acquia Search is a hosted service, it takes away the burden of installation, configuration, and operational duties to keep the software fast, secure and up-to-date. That's our job.

As a reference, we've spent the last 9 months developing Acquia Search with the equivalent of three full-time employees. This also included setting up a billing system, integrating our support system, connecting it to the Acquia Network, performance testing and tuning, and more. Other Acquians helped out with the infrastructure, quality assurance, product management, design, and documentation. It was a non-trivial amount of work.

The result of these efforts is that we can launch any number of Solr farms on Amazon EC2. For high-performance and high-availability, each farm has a master Solr server and one or more slave Solr servers. A load balancer pushes content changes to the master Solr server, which are replicated by the slave servers. The load balancer makes sure that most regular search queries are done against the slave servers. Because multiple servers can handle your site's search requests, Acquia Search is fast and can scale, but it also means that Acquia Search is very robust because it can survive a server failure. As I wrote yesterday, Acquia Search is faster that Drupal's built-in search -- especially for large sites.

In most scenarios, several Drupal sites share a single Solr farm -- by sharing resources, we can offer a high-performance and high-availability search solution to small sites at relative cheap price point. For really big sites, we can provision a dedicated farm and scale out Solr so that it can handle millions of search queries.

Once you begin to use our search service you'll be able to disable Drupal's built-in core search. When you do this you reduce the amount of memory and processing power needed by your own infrastructure. As we've learned with big sites like drupal.org, Drupal's built-in search can bring a large site to its knees. With Acquia Search, you can avoid the drain.

On the front-end, we made significant contributions to the Apache Solr Search Integration modules on drupal.org. We helped add new features, improve the usability, and iron out a legion of bugs that cropped up during the beta period. The top-3 most active maintainers of the Apache Solr module are all Acquia employees, respectively Peter Wolanin, Robert Douglass, and Jacob Singh. As a result, Peter, Robert and Jacob are sometimes referred to as Acquia's three Apache Solr Musketeers.

All things combined, Acquia Search makes it staggeringly simple and low-cost to get better search on your site. You can get started in minutes and you don't have to worry about installing, upgrading, monitoring, or scaling the software. In short, we built an enterprise-quality, highly-available, secure, scalable, and fast indexing search solution that we believe Drupal was missing -- especially for the enterprise.

Acquia Search: benefits for visitors

Why will the visitors of your site care about Acquia Search? For a while now, I have Acquia Search installed on my personal site. To understand what Acquia Search can do for your site, have a look at what it has done for my site. While I have a very simple Drupal site, you should be able to experience some of the benefits of Acquia Search.

For example, search for "Drupal" on my site (use the search widget in the sidebar) and you can see the facets that allow you to filter the results by topic, location and industry. Using these facets, it should be pretty easy to find all the Fortune 500 Drupal sites that I blogged about in 2009, for example. Facets make search faster, making it very easy for your visitor to drill into results and to find what they are looking for.

Acquia Search makes search easier because it is built on the principles of progressive disclosure. Instead of showing the visitor an initial page with lots of complicated options (see Drupal's advanced search options that almost no one uses), the facets are only shown after the initial search query. Plus, and this is really cool, facets are dynamically generated based on the search keywords. As such, they are relevant to what you're searching for.

Acquia Search provides a more powerful search because it is based on the renowned open source Lucene and Solr technologies from the Apache project. Not only do they sport better search algorithms, advanced content normalization, and a "did you mean?" feature, they also come with other great features such as word stemming, document search, range queries and more.

My favorite feature of Acquia Search, at least for use on this blog, is the "more like this" feature -- on node pages you can ask Acquia Search to suggest related content. I have been using it on my site for a while (see the block in the sidebar), and it has helped to keep visitors on my site longer. I occasionally find myself getting side-tracked by the "related links" -- it is a great way to re-discover old posts.

Last but not least, our new service makes for better performance. We performed tests of searches on a Drupal site with over 10,000 nodes of content using a 3.2Ghz dual core server with 1.7 GB of RAM. With Acquia Search results were displayed in less than half a second, whereas the same results served from Drupal's built-in search took anywhere from 1.5 to 7.7 seconds. On the web, faster is better.

That makes for a lot of good reasons why the visitors of your site might care about Acquia Search. Tomorrow, I plan to write a more technical blog post about how Acquia Search works, how we made it that fast, and why it matters to site administrators (instead of site visitors). In the mean time, I recommend that you play around with the search feature on my site or that you sign up for a trial subscription. Have fun!

Acquia Search available commercially

It's a big day for us at Acquia. We finally took the beta-wraps off of Acquia Search, and made it available commercially as part of the Acquia Network. Thanks to the 250+ beta testers who helped make our hosted search service fit for use in production environments, including Brightcove, JackBe Developer Community, P-O-P Design, Wide Divots and others.

We used the beta period to look at the usage statistics, costs, and to talk to a lot of beta users to figure out the best pricing model for this service. We decided on the following:

Acquia Search is included for no additional cost in every Acquia Network subscription. Basic and Professional subscribers have one "search slice" and Enterprise subscribers have five "search slices". A slice includes the processing power to index your site, to do index updates, to store your index, and to process your site visitors' search queries. Each slice includes 10MB of indexing space - enough for a site with between 1,000 and 2,000 nodes. Customers who exceed the level included with their subscription may purchase additional slices. A ten-slice extension package costs an additional $1,000/year, and will cover an additional 10,000 - 20,000 nodes in an index of 100MB.

For my personal blog, which has about 900 nodes at the time of this writing, a Basic Acquia Network subscription ($349 USD/year) would give me all the benefits of Acquia Search, plus all the other Acquia Network services.

For some of you, this might sound like a lot of money, but we believe you get a lot of value in return. In my next couple of blog posts, I plan to outline the benefits of Acquia Search to your site visitors and to Drupal site administrators. Stay tuned!

Acquia Search screencast with RedMonk

In this screencast, Bryan House from Acquia discusses Acquia Search with Michael Coté from RedMonk. Great demo, Bryan!