Posts Tagged ‘rightscale’

Experiences in deployment of my Unity application to Amazon EC2

June 22, 2014

In this post I will document my experiences of attempting to deploy my Unity application to the cloud.

First of all, I decided that I would use RightScale, and Amazon EC2, as I had succeeded in using this combination in the past to deploy smartfoxserver2x-Unity prototypes.  Of course, in this instance, there was a new dimension – interaction with the database.

In the end, I adapted the smartfoxserver template by adding a couple of my own rightscripts.  In particular, I was interested in doing the following things on boot of the server in addition to installation of the SmartFoxServer2X instance:

  • download extension jar files from dropbox
  • download a shell script from dropbox
  • download a mysql representation of my database from dropbox
  • install mysql server 5.5 and start the server
  • load the database file into an appropriate table
  • editing the log4j.properties file for smartfoxserver to allow trace logging (via log4j)
  • editing the xml document for the appropriate room to reflect desired configuration options for the extension jar file (optional, see below)

And finally

  • an operational rightscript that would redownload the database and the extension jar file to the correct location
  • an operational rightscript that would copy the server logs to my dropbox (wput)

Admittedly, the first three are not hyper-secure in terms of data-security, but as I am primarily interested in just getting things to work, and relatively cheaply, this seemed like a good compromise – particularly as for a t1.micro server (that I eventually plan to use, in lieu of an m1.small) – the only way to configure the thing is on boot.  I eventually opted for a RightImage that was an Ubuntu 12.04 machine.

Additional things I needed to do involved:

  • configuration of the smartfoxserver admin to enable the extension jar file
  • management of the appropriate security groups to include ports for MySQL

Eventually I managed to get all of this to work (after much in the way of abortive attempts with seeking to jump straight past MySQL 5.5 to upgrade to MySQL 5.6 which I did not need)*, but still was not able to validate / log into the application. This led me to investigate using log4j within my extension jars to capture / trace behaviour at that end – however it turns out that even this was not necessary, since the server should still have been logging what was going on with the program.

*As a minor aside, MySQL is quite easy to install – indeed, I found that the following was sufficient for my purposes:

export DEBIAN_FRONTEND=noninteractive
apt-get -q -y install mysql-server

At this point, somewhat dissuaded by an apparent lack of progress, I decided to focus on other things for a number of months.  Finally I returned to the problem of cloud deployment of this project relatively recently, and finished addressing the remaining roadblocks.  Essentially the key things still left unfinished were:

  • The fact that the smartfoxserver version was out of date.  I fixed this by altering the rightscale scripts that initially performed a wget on a file in Amazon S3 to a zip file in my dropbox, which I then instructed the server on boot to unpack.
  • UDP configuration in SFS2X/config/server.xml.  It was necessary to provide the server with access not only to TCP but also to the UDP protocol.  I had previously opened up the ports with an appropriate security group but for some reason I hadn’t arranged for the server.xml file to be configured correctly.
  • Basicexamples.zone.xml.  Running locally, I was able to configure this without hazard from the admin page, but due to permissions on the virtual machine this didn’t work properly in the cloud.  I managed to get around this by editing the file manually from the terminal (sudo pico).  In particular what needed to be modified was:
    • Setting the custom login flag to true
    • Adding a reference to the appropriate extension jar class
  • Capitalisation in one of the scripts for creating a room leading to the server not being able to recognise the room extension – I think this was either an operating system thing (less brittle on Mac than Ubuntu), or it might have been rendered less fragile in SFS2X 2.8 (the one I can run locally) but in 2.7 I needed to be more careful.  Regardless, changing the name of the configuration parameter from lower to upper case fixed this issue.

Fixing all these items had the net consequence that, for one thing, I was finally able to see my print / debug statements from my jar files in SFS2X/logs/smartfox.log while the server was running.

More importantly, the end result of all this was that I was able to log in into the world and obtain similar location persistence as I did here.  In other words, I was able to move in the world, log out, and then log back into the world at the same spot, but rather than things running locally, I was connecting from my client to a server and database running on Amazon EC2.  Neat.

__________

So that’s my report as to my experiences with Amazon EC2.  It took a few months, but I got there in the end.

In terms of next steps:

I’ve been giving some thought to how to keep track of the movements of creatures (eg, mobs and npcs) as well as persistent world events (such as weather, day / night cycle) in the application.  This is an issue because although I have managed to obtain persistence of player characteristics (eg, potentially level, stats, equipment, friends list, and, of course, what I have implemented so far, namely position and rotation information) if two players log in and run all the creatures / weather /etc on their clients, then they will have different views of the environment than each other.

To get around this, one would need to have I suppose what might be aptly named as an instance of the game running on the server, with say a vector of transforms / spawn points representing creatures (as opposed to a singleton transform & spawn point for each player), and timers for the day / night cycle, etc, all sent to the server program, which in turn would synchronise with the database and any other player clients connected.  There is really no getting around the need for a separate client running on the server, since creature movements would be too complex to model properly if merely say updated in a java program, for instance.

However, more in terms of a more concrete roadmap and action plan, I am fairly interested in refactoring this project in preparation for Unity5, as then perhaps I might be able to take advantage of Unet, a perhaps more sensible networking framework to use than SmartFox, if I am going to be developing in Unity.  It is also important that I find a good way to simplify my project layout – although I finally have the thing under version control, the project structure is a mess and contains a great deal of assets that I don’t need or am not using, or are not in a particularly sensible structure.  One of the things that I’d like to do, for instance, which sadly is not the case now, is to have a clear separation between source assets and project files, in particular, scripts.

Furthermore, I have been thinking a bit more in terms of a general roadmap going forward.  I’ve decided that aiming to empower players to create dungeons, etc, in a way that is easy to put something together rapidly, might be a good way to go.  I know that Neverwinter Foundry already does this sort of thing, but the angle that I had in mind was to go for a more of a 3D version of Roll20 – where players could have tools / templates to be empowered to create worlds, and run dungeons and dragons sessions in such mockups.  Then the emphasis / focus would be more on providing tools for a good dungeons and dragons experience, rather than so much as building a complicated combat, leveling, and itemisation system.  In this way, I would seek a slightly different path than the one trodden by many of the more formulaic hack’n’slash MMORPGs that are all over the place these days.  Less action, more emphasis on old school role playing.

Anyway, I guess it is something to think about.

Advertisements

Latest developments in the PaaS universe

June 28, 2013

Hi folks,

I’ve been considering the feasibility of maybe setting up a small PaaS on RightScale using Amazon EC2.  There are a few bits and pieces that might facilitate this.  However, most recently, I came across this interesting post, which pointed me to Dokku, a project that is only a few days old, but looks quite promising and very interesting.

An alternative strategy for PaaS cloud hosting might be to write a RightScript or three and get Sandbox installed subsequent to Docker on a virtual box on EC2, via this tutorial (a key difference being that RightScripts would likely be taking the place of Vagrant in setting up the system, if using RightScale).  However Docker itself only supports Python (as yet – of course, the project is still young!), and does not yet (it would seem) support databases, as required with Django apps.  Dokku, however, I believe supports Django and a number of other techs that are not even Python based – and it has some quite clever chaps from MIT, I believe, driving it.

Dokku builds not only on docker, but makes use of a couple of other useful github projects in order to work its magic.

Essentially it should allow the admin to deploy multiple data-driven websites / cloud services (eg, data monitoring / data feed input, then email / sms notify relevant subscribed users, say by google cloud messaging) / applications on the same server, but, according to the project description, it does not yet support multitenancy – which I assume would mean multiple admins / admin user logon (controlled by master admin), not does it yet support multihosts (ie, scaling to multiple virtual boxes, or failover between same).

More here.

Structs in C++, Unity game, and other musings

March 18, 2013

Hello, again,

Yes, as per my previous discussion, it is a bit of a shame that I’ve since lost the posts I had here earlier.  Alas, the perils of not fully testing deployment / upgrade options on a remote service!  Still, the benefits of having this service far outweigh the minor inconveniences caused by the dropping of some data.  And I can still remember most of what I posted.

I’ve looked into the concept of taking backups with dotcloud, I’ve made some progress in this regard, but in terms of the whole backup / recovery approach I think I’d much rather keep hard copies on my computer; so that is what I am now doing.  Certainly using S3 as a backup mechanism gives me the heebie jeebies, since the cost of something going wrong could be a problem.  There are apparently two other approaches that one can take – ftp and ssh.  I may see if ssh is doable at some stage.

In terms of the Unity game I’ve been constructing, the links to the videos in question are first video, second video.

In the first video, I demonstrate logging into the Smartfoxserver 2x FPS example deployed to an Amazon EC2 instance using the RightScale overlay.  I view this as quite an achievement and feel rightly proud of being able to build the project as it stands in this video, even though the code is not my own.  The cost to me of this project was $2 / day, since I deployed on an EC2 small server (I have since decommissioned this, once I had established the utility of the concept).  The cost to use RightScale is nothing for up to 5 of these, since they use 300 of what are known as “RightScale Compute Units”.  One only starts to get charged in excess of 2000 RCUs.  However of course the Amazon instances, or RackSpace, or other service configured with your RightScale account – you have to pay for these.

I kept the default configuration in this example (4 players per game, and a fairly coarse update rate).  Boosting the update rate for the server could lead to a more playable experience (ie less apparent latency / “lag” – in addition to the real latency, of course!). As for the server itself, it can support up to 100 connections at a time – more than this requires a licence from SFS (their profit model). Of course, much more than that might well overload a small Amazon EC2 instance regardless.

The second video demonstrates my progress with pulling the technologies used in the first to a project I have been building, following the BergZergArcade RPG tutorials (quite a resource!).  It shows two screens logged into a local SFS 2X server running on my machine.  In the video I move one of the characters, and, lo and behold, the transform of the character updates on the other screen.  I was very happy with that achievement.

The way that a SFS-Unity game works is essentially one runs the client (which may sit in the web folder of the server, and downloads to the machine of a player if run in the browser – or as a standalone executable), and the client speaks to the extension jar sitting in the server (which is written in java).  The extension jar handles messaging between player clients.  Quite intricate and hard to debug.  But very interesting.  Certainly if there is a more practical way of managing client / server code snippets and source control I’d be very interested in learning of it.

Regardless, I’ve made a bit of progress narrowing down the issue I’m having with animations not syncing (which is apparent in the second video).  Turns out it is an issue with the controller script; I am going to have to rewrite the third person controller script that ships with the standard asset package from javascript to c#.  Apparently it is bad practice to mix the two in a single project, and there is a necessity to modify the script slightly to call functions from auxiliary c# classes.   After that, I think I’d like to try to knuckle through the problem of targeting another player.  Consequent to this I’d like to make mobs for one player the same as mobs for another.  Following that I’d like to start working on the basic combat mechanics, damage, pk, mob despawn, respawn, etc.

On a slight tangent I’ve also started looking at structs in C++.  I’ve found adapting the neural net struct was useful for dealing with higher dimensional tensors.  However this poses problems for anything practical, such as if one wants a mesh of 1000 x 1000 x 1000 with a 3 x 3 tensor, for instance.  Not practical to compute.  Nonetheless, here is the basic idea:

struct coord_t{
int       m_NumInputs;
vector<double> t;
//ctor
coord_t(int NumInputs): m_NumInputs(NumInputs)
{
for (int i=0; i < NumInputs; ++i)
t.push_back(0.0f);
};

};

struct coord_z
{
int       m_NumNeurons;
vector<coord_t> z;
coord_z(int Num1_Neurons,int Num0_Neurons) : m_NumNeurons(Num1_Neurons)
{
for (int i=0; i < Num1_Neurons; ++i)
z.push_back(coord_t(Num0_Neurons));
};
};

Then, in main, one simply declares coord_t p(4), or coord_z q(4,4).  To set a value one writes

q.z[2].t[3] = value.

However, to reiterate, this turns out to be quite inefficient and impractical.  The script will hang at runtime if the arrays are too large – ie there are computability issues.  So my next idea is to instead take a page from the existing book and, instead of allowing say a metric to be assigned to each point in a space so I can compute, say, geodesics via the non-uniform geodesic equation, I would instead take a big array (say only for spatial, 250 x 250 x 250), and then take as final array a coarser subarray (say on average 100 x 100 x 100).  So that the “twisting” of the metric can be arranged by allowing cells in the x, y, z directions to be identified for intervals from 1 up to 10.

In other words, I would have a uniform base array, and then on top of that array pick out a subset of points, to define a non-uniform array containing metric information.  The metric that I had in mind to use as prototype would be the random matrix Rand(0,1) = ((r, r, r), (r, r, r), (r, r, r)), where r = rand(0,1), a function that randomly assigns a value between 0 and 1, provided that the random matrix is non degenerate.  And this is to be computed (effectively) point wise.  And with the convention that the smaller the value the larger the distance between points should be, and that “infinity” or “1/0” is 10 units.

Anyway something to look into.  Evidently I still need to figure out how to represent metric information in terms of construction of non-uniform sub-arrays, but an approximation algorithm certainly seems achievable.  And should be computable.

Consequent to this, the idea would be to apply a finite difference equation to the total array, bearing in mind that the subarray would be the points where the output array is to be read off.

Hello world!

March 14, 2013

Hello,

Welcome to my blog.  Unfortunately I’ve lost a bit of history to this site, since I was a bit too trusting of the specifications for installing plugins to WordPress with the dotcloud-on-wordpress app.  Turns out that it broke with a particular change from PhP 5.3 to 5.4; so with a new machine I lost a few posts.  Regardless, I will attempt from memory to recap on the content of what came before, and, if nothing else, I will be very mindful to learn how to backup mysql databases on this architecture, as per this listing.

Oh, well.  At least I can remember roughly what I wrote (and at least it was only a few posts).  I will attempt to repopulate based on memory in this first post.

Basically, I’ve played with a few compilers and languages over the last few years.  First I looked into C++, then Java, Java3D, cross-over / bridging techs, then libraries for Java (in addition to Java3D – lwjgl & joal), Eclipse, python, pydev…  Then Spring and a few other things.

[wpzon keywords=”java, python” sindex=”KindleStore” sort=”salesrank” listing=”2″]

One of my first objectives was to learn Java3D and build something useful with it, or at least something that worked.  I ultimately found out the utility of the language was limited due to the fact that it required a kit to be installed to run, even for a standalone executable; this has since changed with its incorporation into JavaFX in Java runtime 7.

Then I looked into Flash and a few other techs motivated by enhancing PDF files.

Consequent to this (in 2011 – 2012), I started looking into django.  When I finally got to the point of being able to build working data driven websites, I became interested in deployment.  Google app engine was my first choice, but it turned out, at least at the time, to be clumsy and unwieldy, mainly due to the fact that it relied on a fork of django that did not have a large community backing it (since app engine runs on NoSQL, which is incompatible with standard Django).

Ultimately I found out that dotcloud was better and easier to use for what I had in mind.  I deployed a few sites to it; a simple photo album, the original version of this blog, a personal webpage, a spring site or two, some javafx examples.

But my interests continued to develop and I found myself interested in experimentation with Game Engines.  I ultimately chose Unity, due to the fact that it is relatively indie friendly; it has an asset store and is comparatively easy to get started with.  In terms of deployment, dotcloud was inadequate to host SmartFoxServer 2X instances of Unity3D games, so I ended up gravitating to the RightScale cloud management platform, together with Amazon Web Services (AWS).

I have since made some progress.  I currently have plans to develop my current Unity3D game (which has as base code from a series of tutorials from bergzergarcade) into a freemium multiplayer game on RightScale, although I have, for now, decommissioned the original website on RightScale where I tested the architecture and am testing things locally.

I am also now currently looking into genetic algorithms and neural networks written in C++ (as one does), as well as scientific programming in python.  RStudio is another tool I’d like to get up to speed with, together with the R programming language.

By the by, for those who did not follow earlier, this installation is based on dotcloud, a PaaS that I have found useful for experimentation.