At today's Hadoop Summit in Silicon Valley, Yahoo! announced the availability of the Yahoo! Distribution of Hadoop, a source-only version of Apache Hadoop that Yahoo! uses within its own search engine. Hadoop, of course, is an open source software framework that helps process very large data sets, and is widely used in large-scale data mining applications as well as in search tools at sites like Facebook and many others. For developers and users interested in Hadoop, it's worth noting that the Yahoo! Distribution of Hadoop has been widely tested and developed at Yahoo! for years now, as Eric Baldeschwieler, VP of grid computing at Yahoo, described in detail here. 

Last summer, we reported on Microsoft's acquisition (reportedly for $100 million) of Powerset, which specializes in semantic search based on the open source, cluster-based software framework Hadoop. This acquisition of an open source-centric search company was more strategic than many people realize. Hadoop also underlies Yahoo!'s search engine with its ability to search large data sets quickly, and the acquisition of Powerset may have played a key part in how Microsoft decided to give up its effort to acquire Yahoo!

Of course, Microsoft's big search engine news of the week is Bing, which I've found to have both strengths and weaknesses. Surprisingly, as The Register reports,  Powerset's technology plays only a small part in how Bing works, but what it does in Bing is open source-driven, and interesting.

eWeek reported this morning that Jimmy Wales, founder of Wikipedia, has closed up shop for Wikia, his community-driven search engine. Wikia Search launched in January 2008, and was reported to be the fifth-fastest growing community destination by Nielsen Online in February 2009. The Nielsen statistics seem to have incorporated data from other sites in the Wiki line, however, and reports say that Wikia Search was only drawing 10,000 unique visitors per month.

Wikia Search was a very different animal than traditional search engines, as it substituted search results provided by algorithms for those chosen and ranked by community members. It's certainly an interesting idea -- it's an idea that might actually work exceedingly well under the right circumstances. For now, however, Wales has opted to put work on community-based search on hold and focus his team's efforts elsewhere. He also holds out hope that community search is workable, and vows that when it takes hold, he'll be there in some capacity, actively contributing or simply cheering on the effort.