Archive for January, 2020

Devanuary retrospective

January 31, 2020

General morale

Great! I smashed the sprint that I set for myself, and managed to complete a large number of the cards that I originally put in there.

What went well?

Looking back at the month that was, I think it went quite well from a productivity perspective.

  • I managed to almost achieve the very ambitious goal I set up for myself, getting networking working end to end.
  • I didn’t end up getting exhausted halfway through. Rest days on Wednesdays and Sundays were a very good idea.

What didn’t go so well?

Sometimes I slept in, so I didn’t have time to do much work on the project before going to work. However, that is probably understandable, and I was able to make up the lost time by swapping out for a rest day on one or two occasions.

Questions for myself

  • In terms of in-game level editing, do I want to just use something along the lines of an editable gridmap, or do I want to go straight for something more sophisticated, like Voronoi diagrams inside Voronoi diagrams?
    • I think I should probably architect in the view that I should be allowing for both. However the data model that I use for the world state should be sufficiently flexible that it can allow for both.
  • Is there a risk that the dndsandboxserver might become opinionated in terms of the logic in the dndsandboxclient? How can I protect myself from this sort of thing?
    • This is a risk. Ideally we want to make sure that the dndsandboxserver needs to be sufficiently abstract that it is not dependent on any logic in the dndsandboxclient. If I do this, then at some point down the line I could potentially sell it as a service for people to use.
    • Ideally I should create technical debt cards if I feel that the client logic is leaching into the server logic in any way, shape or form, and then prioritise these cards accordingly in the backlog of things to do.
  • Should I be writing tests for the client and the server?
    • Short answer: probably I should, but due to time constraints, I’m not going to prioritise this work (yes, I know, this is a lousy answer – but at least it is an honest one).
    • Longer answer: in the long run I should probably add unit tests as part of polish before or after shipping an EAP – assuming that that is my objective. Writing unit tests for Godot games is not something I’ve seen anyone do either, so it will require some thought as to how to do. Rails unit tests are of course fairly easy to write – that pattern is well established.
      • Really, it is a matter of the amount of time I have for this work. I would like to be able to get some form of MVP ready first before thinking about unit testing things and maintaining quality of the product. I don’t have customers yet, and certainly I don’t have customers with expectations based on doing business with me for some time, so things like this are not so critical. However, if this is going to eventually be a paid product, I should have something in place to test things – i.e., have documentation as code.
      • This is first and foremost a hobby project that I am doing on my own time, and therefore I am more interested in exploring what I can build in a slightly less disciplined way first, than focusing on resolving technical debt and observing good testing practices from the get-go.
  • What do I foresee as obstacles to shipping a product?
    • I still need to make my apis authenticated. This is an issue. Another is that if I want to make my service a paid product, then I would need to find a way to integrate with a third party credit card transaction handler, like Stripe (I certainly don’t want to be handling credit card information myself, no no no …).
    • I also need to make a long term decision about whether I want to stay on rails. For the first customers maybe it doesn’t matter, and maybe it doesn’t even matter at reasonable scale, but it is worth thinking about.
  • How far to product do I think I am?
    • Well, that is a tricky question to answer! And certainly a loaded question. Considering that I’ve been talking about this project literally for years, it is risky to signpost anything. However, if the rate of progression in this month is maintainable, and I do that once every two months … then:
      • by end March I should have a good idea as to how to handle in game level editing
      • by end May I should have managed to get the rubber hitting the road on that
      • by end July I should essentially have that done – and be onto avatars in the game.
      • by end September I should mostly have the avatar tokens done
      • by end November I should have tokens implemented and be onto permission systems
      • by end January 2021 I should have permission systems done, and be moving on to polishing things up including authentication
      • by end March 2021 I should be halfway through implementing authentication
      • by end May 2021 I should have finished authenticated APIs
      • by end July 2021 I should largely have finished polish on the dndsandboxclient and have that in EAP
      • by end September 2021 I should have managed to integrate stripe with the dndsandboxserver
      • by end November 2021 I should be in a position where I almost have a shippable product,
      • and then, by end January 2022, I should (maybe) be in a position where I can launch the dndsandboxclient as an example of a game on the dndsandboxserver platform
    • So I guess that means I’m two years to product, provided that I continue to steadily plug away at this project! That is a bit disappointing, but I guess at least it is a moderately realistic estimate. Of course, if I manage to worth three months in four, the above estimate could be compressed to 16 months; less than one and a half years from now. Of course, that might not be a sustainable way to work, so 2 years from today, riding on the assumption that I remain moderately productive and focused on this project, is probably a more realistic estimate.
  • Do I have the resources I need for in-game level editing? Should I do some more legwork first before taking on that complexity?
    • I think ongoing research is important. Certainly any code under the correct license that would help me achieve my objectives, or any courses / materials that are free and/or moderately cheap would be a reasonable investment and I should definitely consider getting.
  • Are there any courses that I should take to supplement and / or help enable me to succeed in this project?
    • Yes, well there is certainly GodotGetaway by Canopy Games. That is an excellent course. I also have access to the 1-bit Godot course by Heartbeast, and a few GDQuest courses as well – which I have not yet started due to competing demands on my time. I should work through all of these really if I am serious about “developing my Godot-engine muscles”.
  • Yes, but what about other things other than just courses for Godot?
    • Ideally, yes, I would like to find something decent to read and / or work from examples in relation to hardcore procedural generation – and also how to represent the data from same. Some more leg work and/or ongoing leg work is probably required here.
    • Potentially I would like to stagger months like January “hack months” with “learning months”, where I focus almost exclusively on things like the above. Certainly in January I still did some work on auditing GodotGetaway lectures, so there is no reason that something should be exclusively a learning month or a hack month … probably rather just more of an intentional focus one way or the other.
  • Do I need a break before jumping back into the project again?
    • I think so. I’m certainly not planning to work on this for the first half of February. Certainly by March though I’d like to get back into things again, and then take another break in April.


  • In terms of a two month time horizon:
    • February shall be a rest month / learning month. I’d certainly leave the door open to work on the project in this month, but I’m not going to hold myself to it.
    • March will be a working month.
  • Wednesdays and Sundays worked great in terms of days to rest. I will continue with that going forward.
  • Using Jira was very useful. I will continue with that going forward, and, on any day that I am working, make sure that I review the sprint board first.

Devanuary 31/1: limited progress with handshaking

January 31, 2020

I managed to:

  • Get refresh servers working
  • Format columns correctly
  • Have an ‘on select’ option for a joinable game
  • Was able to send through a call group to join a server with session owner ip and port

However, I was unable to fully instantiate the new player within the running session. I might have another attempt at that tomorrow.

Devanuary 30/1: presenting data within the client

January 30, 2020

Today I:

  • added clients to the servers payload. In the end I decided that the client can decide which clients are inactive based on their last updated_at timestamp; too many requests to the server from clients would be inefficient. Client machines should have sufficient compute for this.
  • managed to display the servername, client count, and can join status for each active session. I set a flag for can join status based on whether clients were < 4 for a particular session (so that one would cap players to 4 for each session).

Still to do:

  • Format columns and data in the itemlist properly. Maybe I could dispense with the labels I created and just add a header row for my itemlist that can’t be selected with the column names I want. I can probably programmatically change the formatting in code of that single row, too – and also prevent players from being able to select it.
  • Add a timer node to the join game room popup scene, so that every 5 to 10 seconds or so another fetch request is made to the servers.json endpoint, and then the server list updated. This would be to address a problem which is that if a client A wants to join a session, and none exist, but then another client B creates a session, that session is not displayed in A’s view. This would solve this problem.
  • If A and B are looking to join a game, and C has a game with 3 players already, find a way to prevent A and B joining concurrently and blowing over the player limit. Maybe this could occur as a consequence of the next step – during the peer to peer connection handshake, if the session host has client limit number of players already (eg 4) then the handshake could be declined and the client returned to the main lobby screen. This would be difficult to figure out though because calls will be asynchronous. There should be a way of implementing a safe ‘unwinding’ of a handshake here though.
  • Implement on_listItem_selected to establish a peer to peer connection with the ip of the session host A of a particular server if B clicks on same from the join room screen. B should then join the game of the session host A.
  • Finally, find a way to resolve the bug where if one clicks Host Game without having an ip resolved, then the game crashes. This should be easy to fix.

Devanuary 28/1: join game room mockup started

January 27, 2020

Today I mocked up a Join Game room popup node in my Lobby scene, and succeeded in fetching a list of servers from the herokuapp while another game was running, and just returning one server.

Still to do is to figure out how to present this data in an itemlist in the popup, and how to wire things up so that a player can actually indicate they want to handshake with the session owner of a particular session.

Also to figure out is whether I want to return all clients with the list of servers (I probably do) and then whether I want to figure out on the client end before presenting this data in the UI as to which and how many clients are still active (I probably do want to do this on the client).

Finally, I would like to be able to send a ‘refresh’ signal to the herokuapp to refetch sessions every so often while the join game screen is active.

Devanuary 27/1: showing only active servers

January 27, 2020

Today I managed to get the herokuapp server to show only the active servers, and /servers/:server_id/clients.json to show only the active clients for that server. I didn’t use active job to do this; instead I set a datetime column in the client and server tables, and then updated this column when an ‘update’ heartbeat was sent from the client to each table individually (only the session owner can send a heartbeat to the server per my logic).

Next, I would like to fetch from the relevant endpoints and display that information to the client in a control node. After that, I would like to wire up that control node and allow the client to join a server.

Unrelated, I would like to only be able to host a server or join a game once the client ip has been returned from the herokuapp, so I will need to implement some way to freeze / lock these options until that information is available.

25/1 Devanuary: slow going

January 24, 2020

Today I realised that I probably don’t need active jobs at all, I just need to provide recently touched servers, i.e.

# GET /servers
def index
# show servers that have only been updated recently
@servers = Server.where(updated_at: :desc)
render json: @servers

However, this doesn’t quite work. In order for updated_at to change, an attribute of the model needs to actually change … so sending a heartbeat won’t cut it necessarily.

In particular, I believe I need a new DateTime field in lieu of ‘is_active’ on the Client and Server models, and then change that instead. After doing that, I can show things that have recently been updated.

In terms of the client code itself, I would also like to send a second heartbeat to update Server model iff the client is the session owner.

However not much progress today. Mainly trying things and not getting very far.

Devanuary 24/1: Defining the ActiveJobs we will need

January 23, 2020

Today I slept in, so not much of an opportunity to get things done today. However I did think a bit about the active jobs I want to create.

Since the client is sending a heartbeat to the server already, each client has ‘updated_at’ set within the system. Therefore, we probably don’t need to mark clients themselves as inactive or active, since the system knows this already – minus updated_at > THRESHOLD_TIME_INCREMENT should provide a good rubric as to whether a client is no longer connected.

However, we would still like to know if servers are active, since ultimately we’d like to expose those to clients who are looking for active servers!

In terms of doing this, a client could query an endpoint on the rails api server, and then this would fetch Servers which are active. How do we know though if a server is active? Well, we could have an ActiveJob that

  • runs only for active servers every minute or so, and:
  • for each of these servers, checks for active clients. If clients are zero, the server is inactive. Mark server as inactive.
  • if there are clients, for each client, call the “clients_cleanup” job. This job 1) checks if minus updated_at > THRESHOLD_TIME_INCREMENT. 2) If so, marks this client as inactive.
  • if there are no more clients left on the server that are active after the previous step, mark server as inactive.

Then, after the outer job has finished running, we have a list of active servers with a count of clients that are connected, along with the ips of the clients that are connected, and whether or not these clients are the session_owner. Return this as a json response to the client.

The client could then present this json response in a Control node i.e. within the UI. The user could then have the option to click on an existing server, and the client should then be able to form a peer to peer connection with the ip of the session owner in order to join the running game.

Devanuary 23/1: Heartbeat implemented for sandbox client

January 22, 2020

Today I:

  • Did a db migration on the server, heroku run rake –trace db:migrate
  • Implemented heartbeat logic on the client, sending “is_active”: true for each client every 5 seconds. I needed to wait to start the heartbeat timer until the world was loaded, of course, since otherwise I’d get that null reference error again. And I realised that http_data needed to be attached to the global Network script.

Tomorrow I intend to:

  • Use active job on the server to poll the clients table every minute or so to see if clients are still active, and mark as inactive those that are not.
  • I’d also like to create a second active job to poll the servers table every minute or so to see if a server still has any active clients, and mark as inactive those that are not.
  • I’d also ideally like this job to be able to be proactively called by clients if their session host disconnects unexpectedly.

Thinking a bit further ahead, it would be useful to store two pieces of game related information in the database too – the player_state, and the world_state.

The player_state is easy – that can be stored as a single field in a table for each session. Not too difficult. We can write this in a fashion that is game independent, and only transmit this small amount of data over the wire from the server every minute or so, say.

The world_state is harder. Because the amount of data is larger, and will potentially accumulate, we don’t want to be sending everything over all at once every minute or so. Ideally, we would probably want:

  • A table to record deltas in the world. One change, one delta.
  • The session owner caches / stores deltas until a packet is transmitted, then sends these to the server in one go.
  • The server then writes this list of deltas as a row in a table say SessionWorldStateDeltas with the session_id as a foreign key.
  • Every so often, the server runs a job to checkpoint the data. i.e., it runs the deltas forward from the default world state to the current world state, and builds a representation in a separate table, with id the session_id. To do this, it will look at rows in the SessionWorldStateDeltas table and make a list of lists of deltas, then run all of these deltas and write a new entry in the SessionWorldState table.
  • Meanwhile, the session owner sends more lists of deltas to the SessionWorldStateDeltas table, until the previous job on the server runs again and writes a second entry for the same server_id to the SessionWorldState table.
  • Then, if we ever want to store the world state and load it, the session owner can call to the server to fetch the latest world state, and, if it is not already cached on the client, it will download it. Then the client can load the world with that state.

The above is probably not required for a MVP but it is probably worthwhile architecting now for what the game could look like going forward.

For the intents and purposes of a MVP, only the player_data absolutely must be periodically synchronised with the herokuapp server.

Devanuary 21/1: database migrations

January 20, 2020

Today I:

  • Added is_active to the Clients table
  • Added is_active to the Servers table
  • Created a new table, ServerState, and established a foreign key relationship with the Servers table, with servers has_one serverstate.
  • Tested seeding for the new structure.

Tomorrow I intend to:

  • Purge the herokuapp database
  • Update the new client and new server creation code to set is_active: true by default
  • Implement a heartbeat mechanism in the sandbox client

The day after that I intend to:

  • Implement a task / resque runner in the sandbox server to mark a client or a server as in_active if updated_at has not been refreshed after some time.

Devanuary 20/1: Taking stock

January 19, 2020

Today seems like a good time to have a brief think about how to proceed further.

I have a couple of key goals remaining for networking:

  • Dispose of sessions that are not running
  • Allow clients to choose and join a running session

Let’s talk about the first one first.

Disposing of sessions that are not running

In order to achieve this, rather than proactively reaching out from the server to each running client as I thought before, it perhaps makes more sense (since I have control over what ports are open on the server), for clients to send a ‘heartbeat’ signal to the server if they are still running the game.

Then the server can mark a client within a session as inactive if it doesn’t receive a heartbeat from it after a given period of time, say 15 seconds.

If the client that is also a session owner is no longer connected to its relevant session, the server can then either:

  • If there are clients connected to this session, make a roundrobin decision to elevate a new client to session owner.
  • If there are no clients connected to this session, mark the session itself as an inactive session after 60 seconds of inactivity.

So it seems clear that we need a few pieces here. The easy part is on the client: the client should periodically send a POST request to an UPDATE endpoint on the server. This UPDATE should include a timestamp indicating when the last liveness signal was sent. This will need a new HttpRequest node in the LocalNetwork scene and a new Timer node in the relevant place, as well as a few lines of code. That’s basically it!

Much of the complexity here in fact will live in the server. We will need to perform two database migrations: one for the Client table, to include a boolean: is_active flag, as well as a timestamp: last_liveness_signal_sent. We will also need a database migration for the Server table, to include a boolean: is_active flag.

Next, we will need to have a worker running on the server, eg a resque worker, to run tasks. One task should be to periodically check that last_liveness_signal for each client on each session is not too old, and mark as inactive those that have not sent signals after a given period of time: 15 seconds, say. This task should also make a roundrobin session_owner change if the session_owner is inactive. Another task should be to periodically check the count of active clients connected to a particular session, and mark the session as inactive if no active clients are connected to it.

Finally, we will need to go back to the client, and on_server_disconnected (not the herokuapp server confusingly, but rather the client instance that was the old session_owner), 1) pause/freeze/halt the gameplay, and 2) hook up a Timer and a HttpRequest to the LocalNetwork scene to poll an endpoint on the ApiServer to fetch the current session, and stop when the session_owner ip address returned is no longer the old session_owner ip address. The new session_owner across clients should then be assigned to the client that has that ip address, and this information synchronised across clients. 3) The game should then be unfrozen, with the new session_owner acting as mediator for network information to be synchronised.*

*More might actually be required here for this to work and information not to get lost. Potentially we would want an additional table on the herokuapp server to be set up to store the game state (transmitted by the session_owner once every few heartbeats), and this information to be pulled down by the new session_owner should the session_owner disconnect. Of course, this is another step towards moving from a non-authoritative to an authoritative server. To set up this new table, all we would want would be an entry for a comma separated array of json objects stored as a string in a single table entry on the herokuapp server, indexed by timestamp and server_id. One catch with this is that the size of this string field would have to be quite large, but varchar can be up to 65500 characters long in Mysql, so that is a reasonable amount of space to play with (alternatively, we could use a blob field instead). Ideally we don’t want the schema of our server to be opinionated with the logic of the game client.

Allow clients to choose and join a running session

This is primarily work on the client. We would like to alter the lobby screen to allow a client to fetch a list of active sessions from the server, by pinging the servers.json endpoint and filtering by is_active: true. The server information should then be listed: server_name, number of players in the session, session_owner player_handle.

The client should then be able to select a session. After selecting a session, a new screen in the lobby should be shown (another control node should be visible, and the old one hidden) and then the client should be able to confirm that it wants to join. (Also, on this screen, the client could enter a password, if/when we allow hosts to set passwords for given sessions. Out of scope for this MVP, though.)

After joining, the client should establish a peer connection with the ip of the session owner. Then, the client should load the world based on synchronised data from the session_owner it has handshaked with. That’s it!

Next steps

So looking at the above, it seems clear that we’ll first need to action the first objective, as the second depends on knowing which sessions are active. However, this gives a concrete plan now to action, and hopefully I’ll be able to make a dent in this new plan before the end of this month.