Archive for March, 2020

Book-keeping server changes actioned

March 30, 2020

Today I actioned changes to the book-keeping server.

  • Installed ruby-2.5.6 locally, had some environment issues
  • Couldn’t test rails locally because of nginx, I need to figure out how to do that later
  • Made the changes to the book-keeping server by actioning a migration (server_ip and server_port columns on servers table).
  • `git push origin master` (github)
  • `git push heroku master` (heroku deployment)
  • `heroku run db:migration`
  • `heroku restart` (so that heroku would pick up the changes!)

I checked using postman. I was thrown for a bit because I thought that I had made all the changes that were required, but heroku didn’t seem to be registering that the columns were present. Even more puzzling was when I run heroku run rails console, Server.connection, Server and saw that the columns were there.

I finally figured out that I needed to run heroku restart for the state of the heroku app to pick up the new columns.

March retrospective

March 27, 2020

General morale

I’ll be honest, not fantastic. I think my morale was at a low ebb over the last few weeks, as I grappled with a fair amount of uncertainty and adaptation to change. And I understand that things will get worse before they get better.

I am very fortunate though, in that I have been blessed in a number of ways, and in a sense I understand that I do not deserve the grace that has been extended to me in these respects. Nonetheless, I am grateful to be able to continue my regular paid work, as it helps to distract me from what else is currently going on. I am grateful for being in a part of the city where I live where it is not too crowded, and there is good access to amenities. I am also grateful for the health of my loved ones and friends, and for the broader communities that I am part of.

My hope is that my local community response follows this pattern, and that leaders in this time of crisis are allowed to step up and take control of the situation. I also hope that this crisis will not go to waste, but will allow people to reflect on things, such as:

  • The perils of complacency
  • The importance of good government
  • The importance of community, and the things that build strong community, such as the church
  • Recognising that we are all connected
  • The understanding of the ephemerality of all things, including our lives, and that it is important to bear this in mind, particularly when considering whether to build bigger barns
  • The importance of respecting expert opinion
  • The dangers of point scoring for shallow political gain
  • The importance of “servant leadership

In terms of more utilitarian things, I was thinking that people might ask themselves:

  • Whether the old patterns of work need be set in stone. Can more of our workforce work remotely / work from home?
  • How they could seed the innovation coming from this unfolding humanitarian disaster into business improvements when the new normal finally arrives – i.e., where are the opportunities here?
  • Can we reduce our carbon footprint in particular ways, bearing in mind the much slower unfolding freight train – with more serious and lasting implications (unfolding over a 200 year time period, and with interest coming due ~ 2030 -> 2070) that is the potential degradation of the habitability of our planet?

What went well?

As in January, so in this month; over the last month I have made leaps and bounds in terms of progress on the project – even though from March 19 I essentially stopped further work for a bit.

  • I discovered the godot_voxel module and how to build procedural terrain essentially from a random seed using that module.
  • I learned a bit about godot’s structure and fooled around a bit with C++, blowing some dust out of the corners of my mind where, long ago, I learned about pointers and addresses for objects in said language
  • I made significant inroads into the Non-Authoritative Networking that I wanted to implement – in particular, I learned about how one could extract the ip of a request from nginx and other things.

Also I did not complete one of my main objectives – that was to allow clients at different ips to speak to each other, I did manage to:

  • Complete the wiring for selecting a server from a list, in-as-much as the client attempted to establish a peer connection to the peer-as-server, albeit on the wrong port. However, I did manage to get the ip correct!
  • I also understood that my naive “just use rails” for Non-Authoritative Networking was not going to work, and was able to successfully architect for an intermediate Node.js server – the Listener – to sit between the Game Client and the Book-keeping Server.

What didn’t go so well?

As mentioned above in some detail, I kind of held it together this month until about a week ago, when my sleeping patterns started to become erratic, and I just decided to focus on self-care for a bit.

However, apart from that (and the evident cause), it is worth calling out the blow-out of complexity of the networking. I do understand kind of why it did – I didn’t realise beforehand that I really wanted to implement some form of NAT punchthrough solution, and after I figured out how to extract ips from nginx after a reasonable amount of engineering effort, I discovered that that solution was not fit for purpose, as the port would shift with each request!

So, in other words, I needed a way to register the port that the server process was using from within the server process. And that meant figuring out how to sandwich a server in between the Sandbox Client and the Book-Keeper that would listen to UDP traffic etc.

The good news is that I figured out roughly how to do that, and even started building prototypes. I also created cards for the next sprint, threw out some things I didn’t need, and was able to readjust my plans accordingly. And this was an excellent opportunity to learn! So not such a bad thing, really.

Questions for myself

  • How far to product do I think I am?
    • My opinion on this has shifted. I think now that I might be able to descope my original plan, and potentially get to product earlier. Particularly now that I am spending more time at home and don’t have a daily 3 hour commute during the week, I have more time, energy and opportunity for this hobby. So:
      • by end April I should be mostly done with NAT punchthrough
      • by end May I should have finished NAT punchthrough and be making inroads on the godot_voxel procedural generation piece for an MVP
      • by end June I should have finished the procedural generation piece, and have added a bit of polish
    • At that point I can probably ship the first MVP!
  • But what about the second iteration?
    • Yes, the second iteration should have a few more bits and pieces. In particular, I’d like to implement:
      • multiple tokens per player (a token is a character object)
      • the ability of the peer-as-server to assign tokens
      • the ability of a player to ‘avatar in’ to a particular token
      • the ability of a player to ‘avatar out’ of a particular token, and merely zip around the map with 3rd person camera
    • This might take me to the end of this year to complete.
  • And the third? =)
    • I’d like probably for the ability to drop particular props into the game as well by the peer-as-server. And probably also a permissions system, where the peer-as-server can allow certain peer-as-client machines to also alter the world. You could probably expect me to be done with that towards the middle of 2021 or later.

In terms of other questions …

  • Have your learning priorities shifted over the last couple of months?
    • I will continue following the Godot lectures for GodotGetaway by Canopy Games.
    • I probably won’t take on other Godot courses at present.
    • I probably won’t do “learning months” staggered between “hack months” as was my plan back in January, but rather learn the minimum I need to continue to advance my project.

Further questions …

  • Do I need a break before jumping back into the project again?
    • Nope!

Actions

  • In terms of a two month time horizon:
    • April will be focused as much as possible on trying to get NAT punchthrough working.
    • May will be mop-up for NAT punchthrough, and I reserve the right to do something else here as well … I guess I’ll see how much runway I have here closer to the end of April!
  • I will try to adhere to Wednesdays and Sundays in terms of days to rest.
  • I will put my health and the safety of those around me first and foremost and reserve the right to step away from this project for a bit if I need to clear my head / help others out etc.

Planning

March 27, 2020

I found it a bit difficult to visualise what I conceptualised in my earlier post of a bit over a week ago, so here is the diagram for the more complicated part, the set up of the initial server:

and of a client joining said server

Things that I started working on were:

  • The Custom Engine
  • The Listener

I have proofs of concept for what is required for both of these. To take the next step and plan out the work, it makes sense maybe to solve for the client-as-server step first.

So…

First of all, in the book-keeper:

  • I need to perform a rails migration on the book-keeper to add columns to the servers table (server_ip and server_port)
  • I need to make sure that these new columns are consumed and rendered by the servers_controller and the servers view respectively.
  • Finally, I would like a post to the servers endpoint to return { id, server_ip, server_port }.

Then, in the listener:

  • I need to allow the listener to take a request with payload of the form `{ type: ‘peer-as-server’ }` (bearing in mind that for the other case that I plan to action later – a client connecting to the server – I will be interested in the listener handling payload with `{ type: ‘peer-as-client’, data: { server_id: <server_id> } }`, which will be handled differently) and then read the ip and port of the requester.
  • The listener should then be able to forward this data to the book-keeper at the /servers.json endpoint
  • From earlier, the book-keeper should respond with `{ id, server_ip, server_port }`. The listener should take this response and itself respectively send an ENet request to the ENet requestee that triggered it reaching out to the book-keeper in the first place. (I should mock the ENet requestee).

Next, in the Custom Engine:

  • I would like to generalise the ENet create_server function so that it will reach out to the listener and send a payload `{ type: ‘peer-as-server’ }`.
  • The create_server function should receive the `{ id, server_ip, server_port }` response from the listener onforwarded from the book-keeper, and expose this as a response to any client that invokes it.

Finally, in the Game Client:

  • I would like to invoke the new create_server function with any additional required parameters that are necessary.
  • Using the { id, server_ip, server_port } response from this function (called asynchronously), I wish to make an HttpRequest to the Book-keeper’s /servers/:server_id/clients.json endpoint to create a client.

I should probably create Jira cards for these … looks like I will need to throw out the existing cards in my sprint and maybe start from scratch with a fresher mindset perhaps at the end of it and the start of the next one, which will be in a few days.

Getting back on track

March 27, 2020

I’ve slept very poorly this week. I will try to get back on track in the next day or so, though, with the project – and my learnings on the Godot Getaway course.

axios & post strategy

March 19, 2020

Today I managed to get post requests working to the herokuapp using the axios library. I also managed to interpret payloads via an example enet client.

However, I still have a problem – when I am ultimately making these requests from my Godot game, I won’t really have access to the server id when making an initial request.

At present, ip/port information is stored in the clients table. clients to servers have a many to one relationship. servers do not have access to this information.

This is fine, but when creating a server, I want to be able to, as mentioned earlier, post the ip and port of the new prospective client-as-server-host to the herokuapp. However, the ENet host is created before sending an http request. I guess I could create the server object via HttpRequest from Godot before instantiating an ENet host, and then pass in the server id to the base Godot engine layer, however that reeks of antipattern … the game engine should definitely not depend on information in a game written for it!

Here is a potentially better approach

  • two new columns in the servers table, “current_host_ip” and “current_host_port”.
  • when a game is started, an ENet server is created
  • as the ENet server is created (the session-host, it establishes a connection to the listener
  • the listener takes note of the ip and port of the ENet server and on-forwards then to the book-keeper to the servers.json endpoint with payload { ip, port } only.
  • The book-keeper responds to the listener with the server_id of the new server record
  • the listener creates a new packet with { server_id, ip, port } information and sends it to the session-host
  • the session-host then knows what server_id it is connected to, and what ip and port it is using.
  • it then makes a request to servers/:server_id/clients.json and sends { server_id, ip, port, …(other params) }

The advantage of doing things this way is that if the session host disconnects, a secondary client could be readily promoted to host without too much difficulty.

We still need to understand though how we record { ip, port } information for clients that are just selecting and joining an active session-host. I would say probably to follow a similar pattern:

  • when the client joins a game with record identifier server_id, it creates an ENet peer
  • while this peer is created, it establishes a connection to the listener
  • we expose a connection to the listener within the game engine api
  • the client directs interacts with the exposed api to send a buffer with { server_id, ip, port } to the listener
  • the listener onforwards the { server_id ip, port } of the connection to the servers/:server_id/clients.json endpoint

Switching again to an npm based package

March 19, 2020

Switching to enet-npm, present in npm here. This is a node.js library with Enet bindings, that has the convenience of being easily installable into a node.js project.

Managed to get a prototype working. The Tutorial.md was misleading … better to just use the examples as basis.

Next step is to make a post request using the data from the client, and send it to the herokuapp. Then I can actually try to connect with the Godot client itself, and test locally. Then, after that, I can try to run this thing online somewhere.

Random wikipedia article!

March 19, 2020

I haven’t been overly productive over the last couple of days.

So here instead is a random wikipedia article on the 1930 Vermont gubernatorial election.

The 1930 Vermont gubernatorial election took place on November 4, 1930. Incumbent Republican John E. Weeks did not run for re-election to a third term as Governor of Vermont. Republican candidate Stanley C. Wilson defeated Democratic candidate Park H. Pollard to succeed him.

https://en.wikipedia.org/w/index.php?title=1930_Vermont_gubernatorial_election&oldid=942654356

So, Stanley Wilson became Governor of Vermont.

A native of Orange, Vermont, Wilson graduated from Tufts University, studied law, attained admission to the bar, and became an attorney in Chelsea. He served in local offices, and was State’s Attorney of Orange County for four years. He was Chairman of the Vermont Republican Party from 1914 to 1917. From 1915 to 1917, Wilson served in the Vermont House of Representatives, and he was Speaker of the House in 1917. From 1917 to 1923, Wilson was a judge on the Vermont Superior Court. He was president of the Vermont Bar Association from 1924 to 1925, and he served in the Vermont House again from 1925 to 1927. From 1927 to 1929, Wilson served in the Vermont Senate. He was Lieutenant Governor of Vermont from 1929 to 1931.

In 1930, Wilson was elected Governor of Vermont. He was reelected in 1932, and served from 1931 to 1935. After leaving office, he resumed practicing law and was involved in several business ventures. He died in Chelsea in 1967, and was buried at Highland Cemetery in Chelsea.

Concerning Chelsea, Vermont :

Chelsea is a town in and the shire town[3] (county seat)[4] of Orange County, Vermont, United States. The population was 1,238 at the 2010 census.

Chelsea is located in a river valley in Central Vermont. The First Branch of the White River travels through the valley and the town. Located in the center of town are two commons

According to the United States Census Bureau, the town has a total area of 39.9 square miles (103.4 km²), of which 39.9 square miles (103.4 km²) is land and 0.04 square mile (0.1 km²) (0.05%) is water.

Here is the location of Chelsea in Vermont:

And here is the location of Vermont in the US:

Switching node.js library

March 16, 2020

I actually need to create an ENet server, so I need a node.js library with bindings for ENet.

Fortunately, there is one here: https://github.com/linluxiang/node-enet

Here is an example server implementation: https://github.com/linluxiang/node-enet/blob/master/echo_server.js#L17-L45

In particular I will want to:

case enet.ENET_EVENT_TYPE_CONNECT:
  console.log('A new client connect');
  var host = enetEvent.peer.address.host;
  var port = enetEvent.peer.address.port;
  makeHttpRequest(railsServer/servers.json, 'POST', { host: host, port: port } )
break;

Creating a node.js listener

March 16, 2020

Today I created a basic UDP server in node.js, a ‘dndsandboxlistener’, based on https://github.com/krystianity/udpws.

Tomorrow I hope to modify godot’s Enet library to do something like the following to connect to said server, after creating a host, within the NetworkedMultiplayerENet.create_server method. (Copy / pasting from a couple of days ago).

/* Connect to some.server.net:1234. */
enet_address_set_host (& dedicatedserveraddress, "bulletphysics.org");
dedicatedserveraddress.port = 1234;
/* Initiate the connection, allocating the two channels 0 and 1. */
dedicatedpeer = enet_host_connect (client, & dedicatedserveraddress, 2, 0);

The first goal is to be able to run things locally and register that a connection has been established between the dndsandboxclient and the dndsandboxlistener.

After this, I’d like to modify the UDP server to send an http POST request to the dndsandboxserver when a connection signal is sent.

Finally, I’d like to deploy the dndsandboxlistener to heroku, make a very slight change to Godot if necessary as well (or allow create_server to take a UDP server hostname and UDP server port as optional method arguments), and then verify that things are still working end to end after doing so.

Last of all, I might create a PR against Godot so that I can push this change upstream so that others can benefit – if I succeed.

Once all the above is done I should have NAT punchthrough figured out, and be able to start moving to polish off the networking functionality. With any luck, I might be able to get to a decent state with it prior to the end of the month.

NAT NAT NAT NAT

March 13, 2020

Try as I might, it looks like ENet doesn’t support posting to a controller endpoint. Just registering to a remote host. Which is almost what I need, just not quite enough. So it is starting to look that direct communication to my rails book-keeping app will not be possible.

I looked a bit more for questions on stackoverflow, and found this one, that gave an example of writing server code using datagrams in Java: https://stackoverflow.com/questions/19087036/udp-holepunching-behind-nat

Looks more helpful in regards to the server end, but it is all in Java. However, it makes me wonder if there is a Datagram library for rails. Early searching led to this medium article https://medium.com/geckoboard-under-the-hood/how-to-build-a-network-stack-in-ruby-f73aeb1b661b .

Doing some more searching made me find that maybe I actually need a UDP server, because UDP / Datagram are one and the same (more or less).

Apparently in 2012 Heroku didn’t support UDP server, but in 2014 they launched a websockets open beta (I guess websockets / datagram / UDP are more or less synonymous or different implementations of the same pattern?) https://blog.heroku.com/websockets-public-beta

More searching led me to the realisation that for a UDP server I might want to use websockets or webutc, and run this in node. Fortunately someone already has a stub implementation of this idea: https://github.com/krystianity/udpws .

So …

  • It looks like I need another server, a node.js server, running on Heroku alongside my herokuapp in rails … in addition to nginx, that now looks extraneous and unnecessary (but which may prove useful later, I suppose).
  • On create_server:
    • The dndsandboxclient can connect to this after creating a host using the ENet library
    • The host can send a packet to the node.js server
    • The node.js server can listen to incoming UDP packets and read the ip and port
    • The node.js server can forward the ip and port by making an http post request with this information to the rails app (the dndsandboxserver).
  • On join_server:
    • The dndsandboxclient pings an endpoint on the dndsandboxserver
    • The rails app returns a list of available servers
    • The dndsandboxclient presents this list in the UI
    • When the user selects a server from this list, the dndsandboxclient then creates a peer for that ip and port using the ENet library

In other words:

For creating a server
For joining a server