Guest Post: How eBay Leveraged Open Source to Streamline Transaction Processing
Performing reliable transactions on scalable websites has become a huge challenge, and every major ecommerce site wrestles with how to keep transactions efficient and fast. eBay, which has staggering transaction processing needs, recently switched to an open source WSO2 enterprise service bus (ESB), with very positive results. In this guest post, Paul Fremantle, co-founder and CTO of WSO2, and vice president of the Apache Software Foundation Synapse Project, discusses why eBay made the decision it did, and what the transaction processing results have been.
eBay Uses Open Source WSO2 ESB to Process One Billion Transactions Per Day
By Paul Fremantle
Enterprises have started to recognize the agility and cost advantages of an open source enterprise service bus (ESB). If they had any last doubts about performance and capability, an implementation at eBay, the world's largest online marketplace, demonstrates that an open source ESB can deliver extremely high performance and high availability in a demanding use-case as well. eBay is using the 100 percent open source WSO2 Enterprise Service Bus (WSO2 ESB) to handle more than a billion transactions per day.
Evaluating Proprietary and Open Source ESBs
More than 94 million active users around the globe flock to eBay to find the best deals in cyberspace. In 2010 alone, the total value of goods sold on eBay was a staggering $62 billion or $2,000 every second. Along with eBay's success comes a huge demand to ensure reliable, 24x7 availability of the services that enable these transactions. There's no room for error, especially during the peak online holiday shopping season.
Back in 2009, eBay assessed whether its custom technology infrastructure could scale to support the company’s rapidly growing business with consistently high performance and reliability. After evaluating, the IT team realized the need for an improved approach.
“The solutions we were using no longer met our needs, so we considered whether to build a new internal system or to adopt third-party technology,” said Abhinav Kumar, senior manager of systems engineering at eBay. “Strategically, we saw several benefits from working with a third- party solution, so we evaluated a number of products on the market.”
A couple of key considerations played into eBay’s decision. First, eBay wanted to accommodate enhanced service mediation and orchestration capabilities to its existing service-oriented architecture (SOA), in order to enhance its business services. Additionally, any solution implemented would need the scalability and performance to sustain the increasing traffic loads of eBay’s fast-growing customer base.
For six months, eBay evaluated several of the industry’s leading hardware and software ESB solutions, looking at both open source and commercially licensed ESBs.
“Open source technology is important as we drive innovation across our platform, and including open source alternatives in our evaluation was a natural fit,” Mr. Kumar noted.
eBay undertook a deep analysis of each competing ESB product that went far beyond a review of the feature sheet. The company reviewed product architectures and completed performance evaluations, often working with vendors to get the most accurate picture possible.
After a comprehensive evaluation process, eBay selected the 100 percent open source WSO2 ESB as the best product capable of handling the middleware requirements of eBay’s online marketplace. The WSO2 ESB, which is built on and extends the Apache Software Foundation Synapse ESB, outperformed all other software options in both speed and reliability. Moreover, unlike ESB hardware, the WSO2 ESB demonstrated the flexibility to grow and adapt to eBay’s evolving requirements for handling transformations, orchestrations, and complex message flows.
Open Source ESB Deployment
Within a few weeks of evaluating the WSO2 ESB in 2009, eBay had an initial test deployment in place that handled roughly 1 million calls per day of live traffic during the 2009 holiday shopping season.
Just one year later, all of the eBay services that are exposed to e-commerce (APIs) are mediated through WSO2 ESBs—handling more than 1 billion calls per day. WSO2 ESBs have also rolled out the WSO2 ESB in internal eBay use cases.
There are now several pools of WSO2 ESB servers, which are deployed in a shared-nothing architecture, running on RedHat Linux and set up on multi-core X86 Intel processors. Each pool of high-end WSO2 ESB servers is provisioning one of several use cases that support eBay’s various business functions: including routing, orchestration, and service chaining. These deployments currently include dozens WSO2 ESB instances, which altogether accommodate external and internal traffic loads from different functional areas such as shopping, trading, checkout, and mobile, to name a few.
The efficient resource utilization of the WSO2 ESB allows massive deployments to run on a minimum amount of servers, saving time and money for the customers. Additionally, the memory usage of the WSO2 ESB instances remains stable irrespective of the traffic load fluctuations at eBay to ensure high availability. Moreover, because the WSO2 ESB instances easily interoperate with eBay’s in-house and third-party monitoring systems, they have helped to improve the overall monitoring capabilities of the system.
Mr. Kumar observes, “Using the WSO2 ESBs, we've been able to provide customers and partners with the quality experience they expect on eBay, even as our global customer base has grown.”
# # #
Paul Fremantle is co-founder and CTO of WSO2, vice president of the Apache Software Foundation Synapse Project, and co-chair of the OASIS Web Services Reliable eXchange Technical Committee.