Common Themes in Scaling
Building a site to scale is still a bit of a puzzle. There’s no magic “scale now” button that works across the board, but there are some commonalities to how successful sites sustain massive growth. The basics of the lessons I’ve been reading through come down to keeping the architecture simple, using commodity hardware or cloud services, and relying on proven open source applications.
Keep It Simple - Geeks in general love to build things, and we love to show off a bit by being clever. Unfortunately, when building at web scale being clever is the exact thing that could land you in trouble when the load starts pouring in. Software that works perfectly in controlled conditions breaks in strange and unexpected ways under load, often in ways that are difficult to track down. Being clever in your architecture could cost you hours in diagnosis if you have unintentionally masked the cause of the flaw. If you cannot explain the setup on the back of a napkin with a few boxes and lines, it’s probably time to rethink your approach.
Avoid “Enterprise” Hardware and Software - Enterprise hardware is a racket, one that should be avoided at all costs. It may seem tempting to design your system around a fancy proprietary SAN for storage, but you will be inviting unneeded complexity into your environment by limiting yourself on licensing and capacity. Proprietary hardware like SANs and special purpose load balancers are expensive, and mostly unnecessary when you can accomplish the same thing on commodity hardware and open source software. Wikipedia, one of my favorite reference architecture sites, released last year that they were using a set of Dell servers configured with 22TB of internal storage each. On the other hand, if building out your own datacenter is simply not in the cards, several sites have seen success with cloud services like Amazon EC2.
Stick With Open Source - The software that powers many businesses, often also labeled “enterprise”, is equally as bad. Closed source, license restricted, and unbelievably expensive, enterprise software will cause more problems than it is worth at some point. Case in point, we once ran our entire stack on IBM’s WebSphere. The databases, the java application server, the web server, and the load balancer. The load balancer used a kernel loadable module that would break every time we patched the server, and we would have to go back to IBM to have a new binary built before we could patched in production. IBM’s turnaround time was normally around a week or so, but for a load balancer, it was completely unacceptable. Own your datacenter, own your software, don’t let a vendor tell you what you can and can not do, leave that up to your imagination.
These are the basic lessons that I’m seeing, keep it simple, own your datacenter (or send it to the cloud), and build out with open source. If you have more tips or other comments, I’d love to hear about them in the comments.