Web applications have relied on relational databases for more than a decade. For the most part, this combination has been convenient for developers; it allowed them to leverage the flexibility and integrity of a database with the agility and distribution of a Web application. However, as sites have grown to handle millions of requests each day, many developers have found that the database can be a bottleneck.
One solution would certainly involve setting up a cluster of database servers; commercial and open-source databases alike are now able to create such clusters. However, clusters are still relatively hard to install and maintain. In addition, they can often be overkill for a Web site whose database is doing many more reads than writes.
Faced with this situation on its LiveJournal site several years ago, Danga Interactive developed the open-source memcached system. Memcached (pronounced "mem-cache-dee") is a, in many ways, the opposite of a relational database: Whereas databases store many types of data in two-dimensional tables that can be interrelated, memcached stores everything in a single key-value table, with rare restrictions on type. A client can connect to a memcached server and either set or retrieve the value associated with a particular key. Because memcached is so much faster than a relational database, it is typically used to cache information that would otherwise have to be retrieved from the database.
For example, many Web sites display a user's name at the top of the screen. This would typically mean that the Web application needs to perform at least one "SELECT" query against the database, to retrieve the user's name. If you are running an online store, then the "shopping basket" page would need to retrieve information from the database about each of the items in your order. Each of these database retrievals might be relatively simple -- but a dozen such requests per page, multiplied by millions of users per day, would slow down even the fastest site.
The solution, then, is for Web applications to first check to see if a needed value is in memcached. If so, then it uses the memcached version, without querying the database. if not, then it retrieves the value from the database, storing this value in memcached. Memcached can also be configured to auto-expire data after a short period of time, to reduce the chance of handling stale data.
Memcached has grown in popularity over the last year or two, both because many Web sites now need to scale, and memcached offers a method to do so that is both easier and cheaper than clustering. But it has also become popular because of a large number of client libraries and modules for various languages. You can now connect to memcached from virtually any language, from C and Java to Ruby and Python. Some application frameworks, such as Ruby on Rails, are now coming with memcached support out of the box, or close to it.
Memcached support is fairly easy to add, and can help to make your Web application much snappier. If your application could use a speed boost, and if the bottleneck is in the database, then you should take a look at memcached, and see if it's the right solution for you.