Guest Post: The Apache Software Foundation's Open Source Approach

ApacheCon, one of the biggest open source conferences of the year, is coming up in Atlanta November 1st through 5th, sponsored by the Apache Software Foundation (ASF). Of course, from Hadoop to the web server, Apache software platforms have become enormously influential. Ross Gardler, VP of Community at the foundation, provided OStatic with a guest post--one of a series we'll be doing in conjunction with ApacheCon--on how the Apache Software Foundation approaches open source. Here it is. 

Apache Open Source - It’s the Way That You Do It

By Ross Gardler, VP of Community, Apache Software Foundation

From the outside, the world of open source could easily be characterised by wrangling over fervent political beliefs on issues of software licensing, patenting and distribution. Despite this impression, for many developers, the big draw of open source software has always been open, collaborative development. Using an Open Source Initiative (OSI) approved license is the basic prerequisite to calling any software ‘Open Source’. However, the OSI’s own homepage starts with the statement: 'Open source is a development method for software that harnesses the power of distributed peer review and transparency of process.'

Within the open source world, the Apache Software Foundation (ASF) has often been characterized by the permissiveness of the Apache Licence 2.0 (AL): the lack of an enforced share-alike clause, the ability to build and distribute proprietary licensed software on top of its open source projects and the licence’s clear and explicit handling if the existing software patent landscape. While this non-prescriptive and pragmatic approach is attractive to many open source projects and developers, it is only part of what lies at the heart of the Apache philosophy and the foundation’s immense success.

To date, half of the top ten downloaded Open Source products are ASF projects (source: https://www.osscensus.org/packages-rank-public.php). In addition to the Apache HTTP Sever, most enterprise Java solutions are developed using Apache build tools and libraries, and Apache projects such as Hadoop, Cassandra, Hbase are already at the core of many of the industry’s most groundbreaking cloud computing and ‘big data’ projects. The common factor behind the success of these and the seventy or so top-level projects at the ASF does not lie solely in the license or the distribution of the software, but in the strength and engagement of the communities built around them.

We are increasingly seeing the emergence of two distinct types of open source software. The first uses an open source licence to distribute the software, whilst development is carried out behind closed doors. Any fledgling community finds it difficult to engage with the development process. The direction, the quality, and indeed, the future of the project are dependent on a single entity, often a private company. In this case, the term ‘open source’ is predominantly used to describe the licensing and distribution method of the software.

For the Apache Software Foundation, open source is more about an open development methodology (ODM). This is characterized by six main traits and, regardless of the strength of the technology and vision, all Apache TLPs have to prove themselves in these key areas:   

•    a deep level of user engagement: if you don't have users then there is no point having a project.
•    transparency: being open about what the community is undertaking and the way decisions are made.
•    collaboration: the ability of working within a diverse group of people, something that the Internet has obviously made easier.
•    agility: once work begins and there is a serious engagement with users, ideas and plans may need to change.
•    sustainability: having the capacity to keep developing an application solution over the necessary period of time.
•    tools: wikis, bug- and version-trackers and email lists to support the development of the community and keep track of intellectual property rights

The roots of ODM at Apache date back to the original development on the Apache HTTP server. Originally a webserver project at the National Center for Supercomputer Applications (NCSA), the project was left in a corner to gather dust when key developer Rob McCool left NCSA in mid-1994. A group of eight concerned users then decided to work together informally on developing and maintaining the project, sharing incremental updates or ‘patches’.

For this group, transparency: making decisions and documenting them in public, became a requisite part of maintaining the project and of building a sustainable development community around it. To this day, openness and transparency are central tenets at the heart of the ASF and the development and direction of all its projects.

Involvement in an ASF project community takes place on one of four roles: users, developers, committers and finally membership of the project management committee (PMC) (often PMC and committer are conflated). Users can be passive or more involved, submitting bug reports and recommendations. Developers are users who chose to submit documentation and code patches to be considered for incorporation into the project. Committers are then those developers who, in recognition of their skills and contribution, have been granted direct access to the project’s repository and can implement both their own changes and those of other developers. Finally, the PMC are group of committers elected by the community to steer the project. Progression through these roles is based on merit and peer group standing. However, all decisions are made on open community mailing lists, thus allowing everyone, including new users, to participate in decision making.

Whilst all committers are able to make changes to the code-base, they are granted this privilege by their peers on the basis that they are deemed knowledgeable and responsible enough to escalate decisions on more important changes to a communal vote. These more important changes are then clearly flagged up and debated on the community lists.

Most importantly, voting is then conducted on a consensus basis. Whilst only the votes of committers and PMC members are “binding”, the opinions of the entire community are heard and a single veto is often enough to prevent a change. Any veto must be accompanied by reasoning to encourage constructive solutions, and silence from the list is taken as consent. In this way, the entire community is engaged and given ownership of the direction of the project.

Consensus does not eradicate the disputes and dissension that happen as a part of any open development process. However, it does necessitate collegial behaviour with the need to convince the entire community of the merit of any changes. This prevents the project from becoming ruled by majority blocks, loosing alienated groups and even contributing to unnecessary project forks. And, in a worst case scenario, where an Apache community reaches an impasse with an inability to reach consensus, voting records give the overarching PMC an ability to step in with a clear picture of where the issues lie and how to act.

Moreover, open voting records also mean that developers coming afresh to the project are not simply presented with a potentially bewildering status quo. Instead, they have an audit trail with the reasoning behind all significant historical decisions on the direction of the project. This, combined with the merit-based governance structure, ensures that newcomers have the opportunity to become every bit as involved in the project as long-term veterans and even the project’s founders. This equality of opportunity is vital in attracting new blood and ensuring the long-term sustainability of ASF projects.

The ASF is by no means the only open source software foundation to use open development. However, for the ASF it is the open development methodology that is at the heart of their vision of what open source is and how they operate.

This article’s author, Ross Gardler, is an ASF Member and VP of Community at the Foundation. Gardler will be speaking at this year’s ApacheCon NA in Atlanta from the 1-5th November. For more information visit: http://apachecon.com.