Archive for November, 2014

Kubernetes, Docker, and Mesos

November 30, 2014

Recently I noticed a couple of interesting new developments emerging in the area of cloud deployment and distributed computing with virtual machines. A while ago, I experimented with dotcloud (in about 2011, 2012) when it was in its early start up stages, and found the tools therein a joy to use and play with setting up (and running) sites based on dotcloud’s dedicated servers. Later on, when dotcloud moved into their production phase, I let that particular venture slide, but found their release of Docker to be a further amazing development.

Docker in and of itself is very powerful, but does not afford the full flexibility and utility that dotcloud’s engine and tooling does, as their tooling allows / allowed one to assign a group of docker machines, with some form of ‘scaling’ factor, and with different groups of machines being assigned to particular microservices driving a web application. So, for instance, one might have a MongoDB database as a backend, and a java service for the frontend to implement a Spring application. Hence I was pleasantly surprised to discover, through reading the 5 year milestone blog entry for the release of the go programming language, that google has deployed a project called Kubernetes (from the Greek, meaning ‘helmsman of a ship’), also on github, that does exactly that – where one has the idea of a ‘pod’ of docker containers, with isolated volumes, and replication controllers, and microservices. All very exciting!

Both Docker and Kubernetes, it turns out, are built on Go, which is a programming language that has been designed and built / optimised specifically for cloud programming. Having discovered these new animals on github that seem to me likely to become systemically pivotal to the next steps in the web’s continued growth and evolution, I did some more reading on projects that have been implemented with Go and discovered the existence of Apache Mesos, which is essentially a way of abstracting computational resources from individual virtual machines, and is a step towards an operating system for distributed computing. Mesosphere on Github is a project consortium whose mission seems to be in line with said vision – and they have a project “Kubernetes-Mesos” that I think would be well worth watching, as well as “Marathon” which aims to be an “init” analogue for multiple VMs on a Mesos cluster, I think, may also become quite pivotal in the months and years ahead.

For a more mainstream take on these matters, there is this article at Wired that might be of interest for further reading on google’s kubernetes initiative. Here is a quote from the article in question:

“It’s part of an arms race. There are literally dozens of tools coming out,” says Solomon Hykes, the chief technology officer at Docker and the driving force behind the company’s software containers. “But Google joining that battle–with code that comes from their massive experience–helps show where this kind of thing will go.”

All quite fascinating.

But what does this mean for everyday folk or interested hobbyists like me or you?  Well, certainly, I would not mind deploying a (very small) Kubernetes cluster using rented cloud resources (eg, on GCE, Amazon EC2, Rackspace, Azure, etc) as part of a hobby project at some point.  The ability to scale is also enormously compelling, particularly for a small player who is interested in managing their OPEX to accommodate growth in their customer base.  More fully blown type implementations, such as incorporating Apache Mesos, are perhaps more suited to bigger players, say corporate type programming / systems engineering.  Of course, the number of corporations that would actually make use of the sort of tooling described in this article in the next one or two years could probably be counted on the fingers of one hand (after all, how many Twitters and Googles does the world have, today?  Not very many, I would wager).

However I am reminded of the apocryphal quotation (circa 1943) that there would maybe be a market for perhaps 5 computers in the world.  So maybe it will be the case with technological innovations such as this as well.  Maybe by 2030, we will see relatively widespread adoption of this sort of thing by large companies, even SME, or even individuals, routinely using 1000s of VMs, as a user of a computer today would not think twice of using billions of transitors today on a single machine.

An open source in-game unity editor

November 15, 2014

I’ve been continuing to think about my aspirations for my Unity game, in particular, empowering players to build dungeons and function as a live ‘gamemaster’ / ‘dungeonmaster’, where rooms can be placed while players are in the game, and monsters / spawns can be dragged and dropped into the game. Also I’m keen to build an application / game where dungeon states can be saved or persisted. Regardless, as I mentioned back in this post, that seemed like an awful lot of work if I sought to build such from scratch, and on further reflection I have since come to the conclusion that it would be far handier to be able to adapt some resource that someone else had built to my purposes, or use some tool that another group had constructed in order to accomplish my objective (rather like I did with SmartFoxServer for enabling multiplayer in my game, with its quite user friendly API and accessible examples).

Hence I was quite excited today to discover that someone seems to have done just that, or at least built the germ of an idea that I and others can use. The project is OpenEd, and it claims to be essentially an open source in game Unity editor. Very cool! The github repository is here.

Certainly the task I have set for myself seems slightly less impossible now. I look forward to a time when I can dust off my old project and start trying to introduce this new functionality, hopefully around Christmas or the new year. I’ve already done a bit of work in simplifying the project structure; what I need to do now I think is to clean up the assemblies (there are about 4 or 5 weirdly named projects when looking at the scripts in MonoDevelop) / or build the scripting again from scratch, and also probably consider switching from MonoDevelop to Visual Studio, the flagship Microsoft IDE, which, in other good news, is now free for developers to use, and which has a Unity plugin for facilitation of Unity game development therein.

Experiences at the Unite Australia conference in Melbourne, 2014

November 8, 2014

Hi folks,

I thought I might write briefly about my experiences at the Unite Australia conference that was held recently at the Melbourne Convention and Exhibition Centre – right next door to the building known affectionately by locals as ‘Jeff’s shed’.

Broadly speaking my first impressions were rather wet. It had been pouring like crazy the night before, and there was an amazing thunderstorm. The trains also were running late. By the time I had made it to the convention centre, I felt mildly drenched, and doubly regretted my choice of formal attire when I noted that all the other folk there – mainly students and young rapscallions in the games industry – were dressed more casually.

It was a fine day. I think the event was sponsored by a few notable players, including the local branches of Intel and Microsoft. Swinburne also made an appearance – they had a booth set up. And there was some graphics card manufacturer as well.

Soon it was time to witness the keynote address, and it all looked quite interesting. Mainly the discussion revolved around the upcoming changes in Unity 5, and the big improvements from Unity 4, mainly due to the new and improved light maps and the way that sound was managed. There was a preview of the new unity user interface system, and it was mentioned how there were performance improvements with that over OnGUI for some reason. There were also a few hints as to the future, with a preview of an evolving prototype slated for the ‘5.x’ release cycle of the Unity engine – this was a feature called ‘Director’ (I think that was what it was referred to as) where one could essentially generate some form of workflow for a game, and things would appear in sequence via some form of state machine, demonstrated by a 2D runner with crates spawning, buildings moving in the background on loop, and various obstacles appearing in a pseudorandom fashion. Other than that, there were a few figures bandied around, such as that the Australia – New Zealand games industry actually does seem to generate a respectable chunk of GDP ($80MM I think), and roughly 70/80% of developers use Unity. And also references to the phenomenal growth of Unity itself over the ten years since its inception, and the responsibility that places on the key developers.

Following the keynote I attended a talk upstairs on UNet, the upcoming multiplayer functionality that I’m quite excited about. The fellow there provided a ‘hack-and-play’ demo where he demonstrated migration of a game written without multiplayer to a game with multiplayer. The philosophy was, that, contrary to the generally accepted wisdom in the games industry where ‘adding multiplayer ‘at the end’ of the release cycle means that you really are only about 50% done’, the new workflow for UNet using Unity really does allow relatively easy migration of a game from single to multi user usecases. However, I guess the questions that I had (that I probably should have asked then, but didn’t really think to at the time) were as to whether UNet allows for a centralised messaging server (like SmartFoxServer or many of the other offerings), or whether it was merely a peer-to-peer / failover type multiplayer shared game. Which is fine, but I like the idea of having a server, as it allows one to control other aspects of a game or application, such as data.

Next I attended a talk on best practices for Unity. This was quite useful to me, in many respects. Performance was a key area of focus, namely the warning to avoid using ‘find’ functions in the ‘update’ part of a unity application, as otherwise things can slow down quite a bit. In particular, calling expensive or clunky code for every mob on every frame can really cause performance issues. Another hint was making use of garbage collection oneself, dependent on whether one wanted to regularly do garbage collection or only rarely, depending on how the game flows.

Other aspects were kind of obvious but often not managed properly. For instance, having one folder for sprites, one folder for sounds, etc – ie a logical structure to the project. Also having logical naming conventions. I guess for me, this is something I still need to get right since my project is a bit of a mess with assets and namespaces not really aligned properly – particularly from multiple migrations from earlier versions of unity.

Finally another best practice was limiting scope in order to ship – in terms of the ‘triangle’ of project management – scope, resources and time. I guess in my case, it might be instructive to put aside my megaproject at some stage and focus on just shipping something, to gain an idea of what is involved.

After lunch, I attended a talk on ‘Unity for Architecture’ – a fork of the Unity game engine that is designed for architects and is a work in progress. It was a fairly impressive demo, wherein they demonstrated import of terrain map files to generate terrain – based on real geodata – and then imported a house from google sketchup next to the grand canyon. But the most impressive part of the demo was the ability to hold the camera / viewport still and iterate for a polished view / preview of a house / home. The resulting images were quite realistic and would definitely be brochure worthy. In fact, I was so impressed with this talk that I attended a follow up talk upstairs afterwards. One person asked whether one might be able to pre-render a path and then ‘fly-through’ a building with the photo-realistic images, and it seemed to me that that was quite an interesting question, although potentially extremely computationally expensive and maybe even impractical with even top-end modern machines.

The last talk I attended was one given by a graphic artist who was not a Unity employee, but still had some amazing experience, having worked on animated sequences for Saints Row 3, Civilisation 5, GTA LA Noire, and Darksiders 2. It was quite interesting to hear about the experiences that the fellow had had working on particular examples, and how he had managed his time and delivered to schedule. Also to hear how at one stage he had run a studio that built 2-D iOS games was fascinating. For me, it was a rare window into the industry.

All in all, I found it quite an interesting conference. Talks that I didn’t go to included one given by a Google engineer on ‘the new world of advertising / monetisation’, there was a talk on the new partnership between the windows store and unity, and there were talks on the new sound system, on the new user interface for unity, on xbox/ps4 game programming using Unity, a talk on the future of Unity, and a few other talks that all sounded fascinating. Unfortunately, I couldn’t go to them all.

I didn’t end up talking to many people in person, maybe a little silly considering that part of the allure of these events is ostensibly to have the chance to network with likeminded souls, but there is certainly always next year. Maybe next year I’ll remember to dress down a little, too.