Archive for April, 2013

Blender experiments / MongoDB

April 29, 2013

Here is a picture that I made following these video tutorials ([1], [2]).

clock

I intend to start looking at these tutorials next on handling splash in blender, by the same person ([1], [2]).

Changing the subject, I’m still very muddled with the MongoDB / mySQL dichotomy, and suspect that I might need to revise my views as stated in my previous post to reflect this.  However I did find an excellent presentation on MongoDB many-to-many relationships, and how the schema model is different, here.

Essentially, this seems to be the key difference of relevance to my investigations:

  • in MongoDB there are no joins, but data is “embedded and linked”

One-to-many is arranged by referencing tables by id, eg:

blogs :
 { author : "Herge",
   date : "Sat Jul 24 2010 19:47:11 GMT-0700 (PDT)"
   comments : [
    {comment : ObjectId("1") } ]}

comments :
 { _id : "1",
   author : "James",
   date : "Sat Jul 24 2010 20:51:03 ..." }

For many-to-many, a similar idea is used, but this time, the linking goes both ways:

products :
 { _id : ObjectId("10"),
   name : "Destination Moon",
   category_ids : [ ObjectId("20"),
                    ObjectId("30")]}

categories :
  { _id : ObjectId("20"),
   name : "adventure",
   product_ids : [ ObjectId("10"),
                   ObjectId("11"),
                   ObjectId("12")]}

Then, to return all categories for a product:

db.categories.find({product_ids : ObjectId("10")})

Or all products for a given category:

db.products.find({category_ids : ObjectId("20")})

Here is an excellent post which describes more or less the same considerations, but in greater clarity and detail.

Actually, it might appear that the problem I am interested in might already have been solved, or close to.  Apparently there is a project called MongoEngine which, according to its webpage, should soon have django support (and already supports flask).  I found that page via this excellent article on how to build an elementary django application, sans admin, using said technology plus a few others.  Unfortunately MongoDB is not the same as the Google Datastore though, which is problematic for myself, although Cloud Foundry might be something to look into in said regard.  Except Cloud Foundry doesn’t support django (although micro cloud foundry, its open-sourced version, apparently does).

So it would seem for now that django-nonrel is the project of choice for use on google app engine.  One-to-many seems supported (through ListField), but not ManyToMany.  The latter would necessitate somehow allowing for db.ReferenceProperty calls via django.models.fields.related.ManyToMany ; ie rewriting that part of the package.   Or, by writing another piece of code, in DjangoToolbox, djangotoolbox.fields.ListPairField, where now the pairs kept track of are pairs of keys, following a remark in this piece on the matter of many-to-many on the google datastore.

Advertisements

Data Structures in SQL vs noSQL – with thoughts to django-nonrel

April 27, 2013

So I’ve been thinking again about the problem of SQL vs noSQL joins.  Basically, for background I am interested in how one could write another python package for the django-nonrel project, such that when a django project asks to do a many-to-many query, running on a nosql backend like MongoDB, the query is sent to that package.  Consequent to this, I would like the package to construct the data structures required to support such on a noSQL structure, ie interpret and build the necessary tables, then manage the interface between django’s data syntax on a standard SQL database and the table mapping so constructed, so that essentially the application is indistinguishable from one merely running on base django, on say a MySQL backend.  Then the behaviour of such would essentially allow one to use any django plugin currently available, without running into the need to rewrite its code (ie, directly alter its associated implicit data model(s)/data organisation, as described in its models.py file(s)).

Hence I’ve done some research, and found the following post on the MongoDB blog.  For comparison, here is a post describing a simple data structure for pure SQL.  Then it would seem that the problem becomes quite clear – whereas it is possible to use the same primary key field name for multiple tables, and join on such in mySQL or SQLite, primary key names must be different for each table on MongoDB or CouchDB, so it is necessary to create a table to associate different primary key field values if joining different tables in same, or some other noSQL backend.  Apparently this is an artifact of the fact that MongoDB must be in third normal form, whereas databases like mySQL are less restrictive, and can be in first, second or third normal form.  Presumably the fact that data must be structured in a certain way in MongoDB, as opposed to the freedom of MySQL or SQLite allows it to scale better, and consequently be the platform of choice for big data type services.

So this reduces the noSQL many-to-many django nonrel problem to the following: given a query that is essentially predicated on the assumption of first (or second, but I think first) normal form, regarding a join, how can one build boilerplate code to do the behind the scenes switch/replace, and create the data structures in third normal form representing the same equivalent information (or query) as per first?

Forum Experiment, part one

April 20, 2013

I now have a database of forums available, at this location: http://myinfo-scs.appspot.com .  This is based on code located here: https://github.com/kjk/fofou and described in more detail here: http://blog.kowalczyk.info/software/fofou/ .

I’ve also been looking at DjangoBB for something similar, in the singular forum sense, from which I could probably then easily abstract to multiple forums.  DjangoBB is rather more polished than fofou, and, of course, is written in the Django framework, which is probably a bit more solid than a free-for-all implementation as above.  The packages I am using, apart from Django Bulletin Board, are:

The price to the latter it would seem is the difficulty in getting the thing to work – in fact, I have since discovered that the only way to make the program work properly is either to rewrite djangobb (hard) or django-nonrel (very hard!).   Although it is certainly instructive to have a bit of a fumble with the djangotoolbox and django-nonrel code, there are inherent limitations with nosql that make a full solution to some of these problems more or less impossible.  Since many of the pluggable components to django (such as djangobb) implicitly rely on relational queries this can make things very difficult for running things on a NoSQL database such as GAE.

There are then several approaches one can then take.  One, is to use Google Cloud SQL – which is not horrendously expensive and I believe can be used for multiple apps for a single instantiation of the API.  Another is to use alternative forum techs, such as fofou, and another which I discovered recently, gforum, though that comes with a user-advisory regarding its hunger for user information via the widget for user logon.  But otherwise relatively promising.  So I am currently looking at defanging that particular piece of code.  For the adventurous/curious, the googlecode repository is here: http://code.google.com/p/gforum/source/checkout .

The third approach is to somehow solve the NoSQL many to many field problem and then incorporate that into django-nonrel.  Apparently this is one of those ‘untouchable’ or overly-ambitious problems in the area of computer science.  But apparently it is possible to solve the problem (although the answer is currently not open-sourced) as per this announcement here (quite recent too, September 2012): http://fatfractal.com/prod/joins-and-nosql/:

I’m an engineer and not usually given to making sweeping statements like, “we’ve solved the many-to-many relationships problem for NoSQL;” but in this case, I hope you’ll agree, it’s merited.

FatFractal are another engine like GAE.  Apparently the engine solves the problem prior to loading to a NoSQL database which I presume is something like MongoDB, such as app engine uses.  Regardless, simply knowing that the problem can be solved wins half the battle.  Indeed, if FatFractal’s claim is true, the fact that the problem is not impossible means that presumably it is only a matter of time before it is independently discovered how to do so, and the knowledge becomes public domain – and thereby applicable to the current django-nonrel distribution / github project (currently at version 1.5 development, 1.4 stable), here: https://github.com/django-nonrel/django/tree/nonrel-1.5-beta .

But until this happens, for me, and other non-experts like myself, I think the best strategy is to use alternative technologies to django (as above) to work on specific applications, since the biggest advantage to django (apart from the built in admin) is the pluggability of components (like ruby-on-rails), and more or less everything breaks and therefore needs to be rewritten (if the data models allow such) for app engine.  Consequently one might as well use web2py which is fully supported.  Although I’d certainly like to learn a bit more about how SQL and noSQL model data ; if nothing else, this would be quite instructive.

As to the question – why NoSQL, if it makes joins so abominably difficult?  The quick answer – speed.  NoSQL is a stripped down version of SQL, and is therefore faster and more horizontally scalable (apparently) so more suited to use with applications / services that need colossal amounts of data (eg, location specific information, weather data).  SQL is slower and does not scale as well, so is more suitable to applications where the data is somewhat more limited, but more highly interwoven and connected (eg, forums, blogs).

Ideas for Unity and forum service

April 16, 2013

In terms of where I would like to take my Unity game, I’ve had another idea that I would like to try.  That is, I would like to have an NPC with a variety of options in a chat window – and that these would transport a player to another scene in the game.  The scene would probably be another SFS room, though not necessarily.  Then certain modes I would like to implement:

  • “Rocks fall, everyone dies” a game in which the player can be the game-master, or where people can team up and be the hapless adventurers.
  • “Double dragon” a game in which the player is the dragon and has to assist an army in crushing another army (AI controlled), or where the player can play as a grunt in one army or another.

For RFED, in game-master mode (vs or against AI) the idea would be to play essentially a 3D game of tetris, wherein blocks would be placed on a 6 x 6 grid, with 10 levels of height.  The AI or players (assisted by AI for empty slots) would then try to dodge the blocks placed by the GM.  The GM would score according to 1. how many players he squishes, and 2. how many complete levels he creates with the blocks (and which thereby eliminate tetris style).

The players & AI meanwhile would be cooperating to activate certain squares in the grid; if a player is crushed the activation would go down by one.  (If a player is crushed they get to respawn in a random location in the grid).  An activation of 4 is necessary to achieve the desired subobjective, which is to effectively raise the height of the board by one, so that the GM has less space in which to deploy 3D tetris blocks.

The game ends when the level has been raised to 10, either by the players or by the GM failing to deploy blocks at height <= 10 (including player level raises).  I would then define victory by assigning and tallying the points in some suitable fashion.

I then thought that the players + AI could have at the GM incorporated in dragon form in phase 2 of the fight, on a flattened version of the 10th level of the tetris grid.  Each player / AI would have control of a “squad” of characters to fight the dragon.  Anyway, just an idea!

In terms of “double dragon”, the idea would be more to have a situation in which there is a massive battle scene AI vs AI and the player’s job is merely to inflict havoc as a dragon.  (I kind of like the idea of using dragon avatars for some reason).

Anyway, these are certainly a couple of goals to aim for in the mid to long term with this project.

————————————

For something of a change of topic, I have been thinking a bit about deploying a forum registration service on GAE.  I’ve decided in particular that an appropriate use of DjangoBB might be the way to go, with the django-registration package to register users – I would then monetise the service using google adwords.  If I became more ambitious along the way, I might introduce a django-admin moderated bannerspace bidding tool; the idea would be for advertisers to bid for “shares” in bannerspace (ie, probability that their ad is displayed), together with specified tags (which would decrease the “share” of bannerspace by restricted the ad to certain sites arranged by tag keyword, but which would allowed the advertising to be more targeted and therefore more beneficial to the advertiser).  Naturally if there was more demand for bannerspace if the bidding tool was set up correctly, so the price would reflect this accordingly.

In terms of the tech I would use for the forum service, I’d probably opt for Django on google app engine, using google cloud SQL.  The cost for a small application seems a bit pricey (~ $1.50 / day for the lowest tier, comparable to what I was spending – $2 / day – on my first RightScale experiment using an Amazon EC2 virtualbox) but there is a free tier usable for one application (0.5GB of storage free), so I might experiment with that.  This, together with the excellent backup options and user-friendliness of the service makes it worth investigating.

Alternatively I was thinking of investigating a service where people could SSH in in an intuitive fashion after setting up an account, and deploy static websites (with ads).  The TokBox experiment I may defer until I am slightly more experienced with these matters.

Regardless, a few things to keep in mind while investigating google app engine as a business tool.

Unity Game – Bridge Battle

April 14, 2013

Hi folks,

So I have a new progression in my “mini-MOBA” series, here.

Previous videos in the series:

In the first video I demonstrated the starter town, avatars, and multiplayer transform sync.  In the second video I took things slightly further and included synchronisation of animations.

In this newest installment I do a few things.  First, I introduce tab targeting.  It is not demonstrated in this video but I can tab between multiple targets ordered by distance.  As part of the targeting procedure I change the appearance of the targeted on the client machine from white to green.  I also display the name of the character.  Finally I display the target’s health in the top right corner of the screen.

Second, I allow for use of the esc key to deselect a target.

Next, I implemented a messenging system (based on the bergzergarcade tutorial #44) on the local client that updated the target’s health bar (and the avatar / prefab’s hp) every time an allowable attack (within 2.5 units, and facing the correct direction) was made on it.  Consequent to this I implemented a secondary messaging system to send the information to the server, and tweaked the server code so that the target’s client would have the updated information for their health bar.  This is demonstrated in the video by my pitched bridge “battle” where as you can see hit points are being decreased over time on the other screen when I make an attack on one client relative to the other.

Finally, I made a hodgepodge attempt at attack animations.  This entailed alteration of the state machine for the thirdpersoncontroller script.  The results were not perfect; the entire animation does not play prior to returning to CharacterState.Idle and the animation synchronisation information has not been transferred.  These are minor matters to be fixed.

Also another bug which is not shown in the video, was that I haven’t altered my code to allow for the logoff or disconnect of another player, which might cause difficulties.  I will deal with this matter prior to the next step in the project, or eventually, anyway.

Learnings from this work were firstly, the DoNotDestroyOnLoad for a gameobject – I found out that all the Start() functions of gameobjects are called even in scenes that have not been loaded yet.  I guess the alternative is to use an Awake() function?!, but I found that one of my gameobjects was being initialised and then being destroyed when I loaded the very scene that it was in! – very frustrating.  But when I found out what object it was (the health bar) the fix was surprisingly easy.

A second learning was the power of the statemachine in terms of driving object behaviour – this is still something I need to become more adept in.

Thirdly, it was very informative to really examine the workings of the server code, in terms of introducing my own messaging functions to implement health synchronisation / modification consequent to a melee attack.

Next on the agenda, after fixing the bugs:

  • fix the attack animations
  • allow for player logoff / dc

These would be:

  • Allow mouseover target skin modification and mouseclick targeting
  • Respawning (destroy playerobject & recreate when appropriate)
  • GameHUD update
  • Implement NPCs as “SpawnNPC” script managed by the server (don’t want multiple copies of same NPC per client instance!)
  • Implement inventory for players
  • Implement “shops” – I probably will not require currency at this stage but just allow them to be storehouses / other inventory screens
  • Allow for player chat (global)

Then, further afield:

  • Allow for conversation with NPCs
  • Monsters
  • Quests
  • Advanced HUD (mini-map in top right hand corner)
  • More trees, buildings, environmental features and places to explore
  • Leveling system
  • Character screen
  • Quest log
  • Inventory screen
  • Stat system
  • Item buff system
  • Basic missile combat
  • Basic magic combat
  • Currency (game) for items from NPCs
  • Cooperation / group adventuring
  • Arena zones only for player-player combat (default atm is global)

And in the very distant future (subject to a great deal of change):

  • Freemium implementation
  • Special effects for magic / melee
  • Special moves (melee eg whirlwind, charge)
  • Special ranged attacks (fireball, lightning, …)
  • Customised key bindings
  • Customised blender models and animations …

Anyway I’ll see how I go with this.  Very early days as yet.  I might well return to the bergzergarcade tutorials if my progress starts to slow considerably.  Alternatively there is something to be said for putting this project on hold, and instead looking again at Google App Engine.  I have a few ideas there that I’d like to investigate further.

dotCloud is going OpenSource

April 13, 2013

Hi folks,

I have now moved my blog to this location, since dotCloud’s Sandbox mode is now moving to an OpenSource implementation, which I have to say is quite commendable of the chaps.  Since I can’t currently afford to maintain my own servers on a 24/7 basis, I consider this an affordable low maintenance solution.

However this does lead me to become more motivated in order to actually build a live application on dotCloud that meets their desired criteria: “Our core competency is and will continue to be the operation and support of large-scale cloud services, for tens of millions of visitors, 24 hours a day, every day.”  Certainly something to aim for!  I’m not entirely certain the nature of a service that I would like to build, but, while I’m casually brainstorming:

  • it could be a service for building rooms for online collaboration, via manyuser-video conferencing, leveraging off TokBox; I would charge some nominal (small) fee for the service on an hourly basis, and then pay TokBox a portion of the take – since TokBox is fairly cheap (I think $200 / 250 for the first 75000 hours of usage per month total), this doesn’t seem like a poor strategy.
  • an online wiki.  Running on mediaWiki, I could host a not for profit specialist wiki focusing on some area that wikipedia might not focus on in the depth that I might like – eg a research wiki.
  • a forum registration service.  Running multiple user forums, the profit model would be advertising revenue and banner space.
  • a game.  I had difficulties with doing this before, however, in terms of where I wanted to take it.
  • an online provider of digital course content that I create, or
  • a provider for MOOCs (massive open online course) that simplifies the process of creating an online network of materials for students to use in their courses.

My impression is that the first and third points are probably the ones I’m most likely to want to try.  And the third would probably be the best to get started with.

The first step to enact the forum idea in particular might to consider Django as the technology to approach the matter.  I know that the lightbird tutorials cover forums in this respect, so I would not be starting from scratch.  It would then be simply a matter of extending / abstracting away from a single forum to creating a database of forums.

I guess in terms of how I would go about testing prior to implementation is that I would deploy on a Sandbox server, self-hosted, then contact dotCloud should I mean to move to Live / Enterprise mode.  Broadly speaking though, I think consideration of a live service is really only worthwhile if one has a passion or driving need to work in an area and sees an opportunity for improvement.  And largely speaking I’m learning a bit more fooling around with Unity currently! and see no obvious ways to simplify online transactional foolishness.  Learning is perhaps the most important thing for me at the moment – knowledge is, afterall, the best enabler of serendipity.

Further progress with Unity Game

April 10, 2013

Hi folks,

I’ve made a bit more progress with my Unity game, as per this video here, which I recorded using QuckTime.  Essentially I’ve succeeded in getting three animations to sync across the server to other player screens – an idle animation (when the avatar(s) are still), a walking/running animation, and an ‘axe-slash’ animation (attack & jumping).

In the video I also demonstrate how the game can theoretically support more than 2 players at once – to be more precise, I show three (although I haven’t figured out how to use the same browser to host more than one login at once without crashing – it may be something silly that is port-related or somesuch).  I am using a Chrome window, a Firefox window, and the Unity development window.

Finally in at least part of the video I show some of my method, in terms of debugging within the server trace – when a message for ‘GoodIdle’, ‘GoodWalk’, or ‘AxeSlash’ is sent to the server it pops up in the monitoring window.  I found that invaluable when figuring out how to get the thing to work – in particular I discovered that the reasons I wasn’t able to get the Idle animation to play at first was because I required that the velocity of the avatar be precisely zero to trigger the change in the state machine, and this became clear when ‘GoodWalk’ was the last message passed to the server according the the log, even when on my client the avatar was still.  Sure enough, I found that on another client, the walking animation of the other avatar was still looping on the spot, which was not what I wanted!  The reason for the bug was because computers are approximate in floating point calculation so this was never going to work – instead I found that setting the absolute speed to be less than 0.1 units for the Idle animation to trigger was what was required.

Consequent to this success, my next order of business will be to determine how to figure out remote targeting, ie targeting of another player’s avatar by server messaging, as well as figure out how to build global chat into the game.  Consequent to building global chat I will see if I can construct a local analogue, ie, that can only be read within a limited range of the individual in question.

Then, following that order of business, I would like to handle combat damage, respawning, and combat / noncombat zones.  And also allow for player groups.  Wish me luck.

Backing up – the simple approach

April 6, 2013

So it would appear that I’ve been mildly silly and overthought the problem of backing up my files.  There is an export / import facility built into wordpress.  Although certainly I suppose knowing a bit about ssh, dotcloud and Amazon S3 could serve me well later, should I be considering operations requiring larger amounts of data and consisting of quite possibly more than one individual, which might well necessitate finer level control over disaster recovery.

This screenshot tells all:

Screen Shot 2013-04-14 at 5.03.46 PM

I’ve also discovered that you can get a plugin, Online  Backup for WordPress, that takes weekly snapshots of the entire blog.  So this looks like a workable approach.

Using the import / export tools, I was able to restore the blog to this location: http://blog2-cjg.dotcloud.com .  The only difference is that I do not have all my plugins!  So, for posterity, here is the list I am using:

  • Jetpack by WordPress.com
  • Online Backup for WordPress
  • SyntaxHighlighter Evolved
  • WpZon Plugin for WordPress

Unity on Facebook?

April 5, 2013

An interesting development.

I wonder whether this will be free with Unity 4.1 – and how the billing and publication arrangements with Facebook will work.

Certainly this has the potential to simplify the specfics of Unity game deployment significantly.