Map/Reduce with Bash - The Unix Philosophy in Action

At the recent Software Freedom Day 2014 event we presented on an interesting solution we developed to a particular "business" problem we were ask to assist with. At the core the issue was how to create a map/reduce framework using only bash and command line tools for a 900 node cluster.

The Unix Philosophy

For us this became a stunning example of the power of the Unix design philosophy. The "Unix philosophy", as it is known, is an architectural approach adopted by the creators of Unix in the early years of the development of that family of operating systems which emphasizes a modular design, the creation of  single purpose utilities and tools which can be combined in different ways to produce solutions to many diverse problems.

Bash and Linux Utilities Combine for an Awesome Solution

In our proof-of-concept solution we eventually made use of GNU parallels, parallel ssh, Linux capabilities and cgroups. Cgroups, Linux capabilities and name spaces are features of the Linux kernel, some of which have been around for years, but are only now beginning to realise their potential, which will propel a whole generation of innovation on the Linux platform going forward. Docker is the one current example of this.

We indeed live in interesting times!