Posts Tagged ‘github’

Takeaways from a recent spark meetup

December 9, 2015

Recently I attended a Spark meetup in my home town.  I learned about a few things.   In no particular order:


Kubernetes, Docker, and Mesos

November 30, 2014

Recently I noticed a couple of interesting new developments emerging in the area of cloud deployment and distributed computing with virtual machines. A while ago, I experimented with dotcloud (in about 2011, 2012) when it was in its early start up stages, and found the tools therein a joy to use and play with setting up (and running) sites based on dotcloud’s dedicated servers. Later on, when dotcloud moved into their production phase, I let that particular venture slide, but found their release of Docker to be a further amazing development.

Docker in and of itself is very powerful, but does not afford the full flexibility and utility that dotcloud’s engine and tooling does, as their tooling allows / allowed one to assign a group of docker machines, with some form of ‘scaling’ factor, and with different groups of machines being assigned to particular microservices driving a web application. So, for instance, one might have a MongoDB database as a backend, and a java service for the frontend to implement a Spring application. Hence I was pleasantly surprised to discover, through reading the 5 year milestone blog entry for the release of the go programming language, that google has deployed a project called Kubernetes (from the Greek, meaning ‘helmsman of a ship’), also on github, that does exactly that – where one has the idea of a ‘pod’ of docker containers, with isolated volumes, and replication controllers, and microservices. All very exciting!

Both Docker and Kubernetes, it turns out, are built on Go, which is a programming language that has been designed and built / optimised specifically for cloud programming. Having discovered these new animals on github that seem to me likely to become systemically pivotal to the next steps in the web’s continued growth and evolution, I did some more reading on projects that have been implemented with Go and discovered the existence of Apache Mesos, which is essentially a way of abstracting computational resources from individual virtual machines, and is a step towards an operating system for distributed computing. Mesosphere on Github is a project consortium whose mission seems to be in line with said vision – and they have a project “Kubernetes-Mesos” that I think would be well worth watching, as well as “Marathon” which aims to be an “init” analogue for multiple VMs on a Mesos cluster, I think, may also become quite pivotal in the months and years ahead.

For a more mainstream take on these matters, there is this article at Wired that might be of interest for further reading on google’s kubernetes initiative. Here is a quote from the article in question:

“It’s part of an arms race. There are literally dozens of tools coming out,” says Solomon Hykes, the chief technology officer at Docker and the driving force behind the company’s software containers. “But Google joining that battle–with code that comes from their massive experience–helps show where this kind of thing will go.”

All quite fascinating.

But what does this mean for everyday folk or interested hobbyists like me or you?  Well, certainly, I would not mind deploying a (very small) Kubernetes cluster using rented cloud resources (eg, on GCE, Amazon EC2, Rackspace, Azure, etc) as part of a hobby project at some point.  The ability to scale is also enormously compelling, particularly for a small player who is interested in managing their OPEX to accommodate growth in their customer base.  More fully blown type implementations, such as incorporating Apache Mesos, are perhaps more suited to bigger players, say corporate type programming / systems engineering.  Of course, the number of corporations that would actually make use of the sort of tooling described in this article in the next one or two years could probably be counted on the fingers of one hand (after all, how many Twitters and Googles does the world have, today?  Not very many, I would wager).

However I am reminded of the apocryphal quotation (circa 1943) that there would maybe be a market for perhaps 5 computers in the world.  So maybe it will be the case with technological innovations such as this as well.  Maybe by 2030, we will see relatively widespread adoption of this sort of thing by large companies, even SME, or even individuals, routinely using 1000s of VMs, as a user of a computer today would not think twice of using billions of transitors today on a single machine.

An open source in-game unity editor

November 15, 2014

I’ve been continuing to think about my aspirations for my Unity game, in particular, empowering players to build dungeons and function as a live ‘gamemaster’ / ‘dungeonmaster’, where rooms can be placed while players are in the game, and monsters / spawns can be dragged and dropped into the game. Also I’m keen to build an application / game where dungeon states can be saved or persisted. Regardless, as I mentioned back in this post, that seemed like an awful lot of work if I sought to build such from scratch, and on further reflection I have since come to the conclusion that it would be far handier to be able to adapt some resource that someone else had built to my purposes, or use some tool that another group had constructed in order to accomplish my objective (rather like I did with SmartFoxServer for enabling multiplayer in my game, with its quite user friendly API and accessible examples).

Hence I was quite excited today to discover that someone seems to have done just that, or at least built the germ of an idea that I and others can use. The project is OpenEd, and it claims to be essentially an open source in game Unity editor. Very cool! The github repository is here.

Certainly the task I have set for myself seems slightly less impossible now. I look forward to a time when I can dust off my old project and start trying to introduce this new functionality, hopefully around Christmas or the new year. I’ve already done a bit of work in simplifying the project structure; what I need to do now I think is to clean up the assemblies (there are about 4 or 5 weirdly named projects when looking at the scripts in MonoDevelop) / or build the scripting again from scratch, and also probably consider switching from MonoDevelop to Visual Studio, the flagship Microsoft IDE, which, in other good news, is now free for developers to use, and which has a Unity plugin for facilitation of Unity game development therein.