OSS Developers and the Road Less Traveled

by Sam Dean - May. 02, 2008Comments (0)

Is it a wise career move for software developers and asipring ones to establish a concentration on open source development tools and applications? Amanda McPherson argues that it is, especially since more and more corporations are using OSS platforms and applications. I tend to agree, although as is true with all investments made toward career advancement, it's wisest to tread carefully. In particular, expertise with platform software, especially emerging platform software, looks to have more and more market value.

Of course, there are those who mistakenly argue, as McPherson notes, that "open source is for hobbyists and time off from real jobs.тАЭ That doesn't imply that open source expertise has much market value at all. Data and comments I'm seeing from the field argue against this wrongheaded thinking. In fact, I might advise a young aspiring software developer to become a specialist in certain types of open source projects--to deliberately stand far from the developer masses.

Last night, at GigaOm's Hadoop meetup I saw this in action. Hadoop is an open source software framework, sponsored by the Apache Software Project, designed to take advantage of huge clusters of computers to produce fast results for queries and more.

So far, the highest-profile applications for Hadoop have been search-oriented. For example, Yahoo's search infrastructure relies heavily on thousands of machines working on Hadoop-driven tasks. However, experts from Yahoo last night made clear that many kinds of tasks that are pattern-recognition in nature can benefit from Hadoop.

The clincher came when Eric Baldeschwieler, VP of grid computing at Yahoo, displayed a chart showing how disproportionately more work Yahoo is doing on the Hadoop project itself than others are. "We're hoping to see more community collaboration, and hope to see people posting Hadoop tools and participating," he said. "We're hoping we can hire some of the people producing these."

Now, I would wager that relative to the number of young, ambitious developers out there who are specialists in Linux, there are far fewer who are specialists in Hadoop. Yet Yahoo is hardly the only organization making extensive use of Hadoop.

Chad Walters, director of engineering at Powerset, was also on hand last night. Powerset does the search for the English version of Wikipedia, which requires many multi-core systems working on distributed tasks concurrently. Hadoop is the key driver of tasks toward this end.

Sure the list of companies who can use skilled Linux jockeys is longer than the list using Hadoop. You can see a list of the number of companies contributing to the Linux kernel alone, here. But there is often market value in being a specialist at something relatively obscure--a big player in a small pond. The folks at the Hadoop event last night made clear that there is opportunity down the open source roads less traveled.