Kubernetes, Docker, and Mesos

Recently I noticed a couple of interesting new developments emerging in the area of cloud deployment and distributed computing with virtual machines. A while ago, I experimented with dotcloud (in about 2011, 2012) when it was in its early start up stages, and found the tools therein a joy to use and play with setting up (and running) sites based on dotcloud’s dedicated servers. Later on, when dotcloud moved into their production phase, I let that particular venture slide, but found their release of Docker to be a further amazing development.

Docker in and of itself is very powerful, but does not afford the full flexibility and utility that dotcloud’s engine and tooling does, as their tooling allows / allowed one to assign a group of docker machines, with some form of ‘scaling’ factor, and with different groups of machines being assigned to particular microservices driving a web application. So, for instance, one might have a MongoDB database as a backend, and a java service for the frontend to implement a Spring application. Hence I was pleasantly surprised to discover, through reading the 5 year milestone blog entry for the release of the go programming language, that google has deployed a project called Kubernetes (from the Greek, meaning ‘helmsman of a ship’), also on github, that does exactly that – where one has the idea of a ‘pod’ of docker containers, with isolated volumes, and replication controllers, and microservices. All very exciting!

Both Docker and Kubernetes, it turns out, are built on Go, which is a programming language that has been designed and built / optimised specifically for cloud programming. Having discovered these new animals on github that seem to me likely to become systemically pivotal to the next steps in the web’s continued growth and evolution, I did some more reading on projects that have been implemented with Go and discovered the existence of Apache Mesos, which is essentially a way of abstracting computational resources from individual virtual machines, and is a step towards an operating system for distributed computing. Mesosphere on Github is a project consortium whose mission seems to be in line with said vision – and they have a project “Kubernetes-Mesos” that I think would be well worth watching, as well as “Marathon” which aims to be an “init” analogue for multiple VMs on a Mesos cluster, I think, may also become quite pivotal in the months and years ahead.

For a more mainstream take on these matters, there is this article at Wired that might be of interest for further reading on google’s kubernetes initiative. Here is a quote from the article in question:

“It’s part of an arms race. There are literally dozens of tools coming out,” says Solomon Hykes, the chief technology officer at Docker and the driving force behind the company’s software containers. “But Google joining that battle–with code that comes from their massive experience–helps show where this kind of thing will go.”

All quite fascinating.

But what does this mean for everyday folk or interested hobbyists like me or you?  Well, certainly, I would not mind deploying a (very small) Kubernetes cluster using rented cloud resources (eg, on GCE, Amazon EC2, Rackspace, Azure, etc) as part of a hobby project at some point.  The ability to scale is also enormously compelling, particularly for a small player who is interested in managing their OPEX to accommodate growth in their customer base.  More fully blown type implementations, such as incorporating Apache Mesos, are perhaps more suited to bigger players, say corporate type programming / systems engineering.  Of course, the number of corporations that would actually make use of the sort of tooling described in this article in the next one or two years could probably be counted on the fingers of one hand (after all, how many Twitters and Googles does the world have, today?  Not very many, I would wager).

However I am reminded of the apocryphal quotation (circa 1943) that there would maybe be a market for perhaps 5 computers in the world.  So maybe it will be the case with technological innovations such as this as well.  Maybe by 2030, we will see relatively widespread adoption of this sort of thing by large companies, even SME, or even individuals, routinely using 1000s of VMs, as a user of a computer today would not think twice of using billions of transitors today on a single machine.


Tags: , , , , , , ,

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: