Current learning progress

February 15, 2020

This week, I audited the last part of lecture 2, as well as lectures 3 and 4 of the Topos institute’s lecture series on programming with categories . I have also started lecture 5.

I’ve also purchased a couple of Udemy courses on Scala (another functional programming language which happens to be more applicable to my current working situation). I will aim to start those courses at some point … maybe not this month, but soon.

I’ve also found out (from lecture 3) that there are these things called small proof languages. Coq is an example of a thing of this type. Very heavily paraphrasing from the lecture:

[Coq is useful in its way for providing proofs of termination of a partial function, but not really practical for most purposes in programming]

and

[Proofs are not a first class object in Haskell (but are in Coq)]

In particular:

Coq is not unique. For instance, the redprl project (a successor to nuprl, another proof system), can be found here http://www.redprl.org/en/latest/

Small proof languages can be thought of as a subset (subcategory?) of functional programming languages, and you can write things like web servers in them, eg https://github.com/coq-concurrency/pluto.

Learning month!

February 10, 2020

This month I’d like to focus on learning a few things. My target will be 2 lectures per week from the Topos institute’s programming with categories lecture series, and I will aim to work through two problem sets from this course prior to March.

Concurrently with this I intend to do some pre-work on procedural generation with Voronoi tilings / diagrams within Godot, and also continue to follow the Godot Getaway series of course videos.

As a stretch goal I’d like to try to familiarise myself with some of the Topos theory blog posts on the Azimuth blog, and also concurrently read Sheaves in Geometry and Logic, but I have a feeling that I won’t really get around to that.

Opportunities, challenges, solutions

February 1, 2020

One of the most important challenges at the moment for global civilisation is the problem of ensuring that our life support systems don’t inflict any great setbacks on us in the next 20, 50, and 100 years (2040, 2070, and 2120). I believe it is meaningful to look into these windows of time as they represent the near, far, and very far future. Therefore it makes sense that the landscape of risks and opportunities will largely be different, driven largely by changes in technology, but also in the way that technology has altered the balance of economic clout in the world.

The short is that I am optimistic that global civilisation will be able to persevere through the years ahead. We have a good story to date, and the pages being currently written auger well, I believe, for the years ahead. Let me now proceed to convince you.

Before I look ahead though, it is perhaps instructive to first look to the past.

1920 – 1970: Business as usual

Although 1930-1945 could be viewed as a series of setbacks, viewed holistically over this period the global economy grew tremendously. There were multiple developments in science and technology. Flight moved from being a curiosity to the point where humans landed on the moon and planted a flag there. A real computing machine following the schematics of Charles Babbage and Ada Lovelace was created. Motor cars became commonplace. Whitegoods such as fridges and microwaves became available. The TV was created. Radio became a commonly accepted commodity. Tremendous strides were made.

The world population also grew tremendously in this period. Times were good.

It is perhaps worth noting that the church was in decline in this period in the West, and had been for some time. This was marked by the protracted period of uncertainty that was to come. Largely speaking, at this point ‘the West’ was largely carried by the inertia of what had come before it, but there would come a time when people would recognise the value of investing once again in social capital.

1970 – 2000: A wake up call

By the 1970s, largely due to the work of Charles Keeling, it became accepted that ppm of CO2 in the atmosphere was increasing at an alarming rate. This precipitated a call to action around the world to solve the looming climate crisis.

The engine of the economy continued well in this period. Advances continued to be made. Computers were miniaturised and became consumer commodities. The world wide web, from its origins in arpanet became generalised to civilian usecases and available to all. Even in the early days of the web an explosion of activity and development presaged how useful it would be in the coming decades as an engine of economic growth and generator of social and technological capital. Mobile phones were developed. No more landlines necessary!

By the end of the 1980s the world had entered a period of peace and overall prosperity, but there was a lingering sense of foreboding regarding the Keeling Curve. Many people laid the groundwork in this period for work on sustainable technologies, like solar cells, wind, geothermal, and other similar technologies, in anticipation that R&D into these areas would be necessary to mitigate a looming climate crisis.

A large amount of progress was made, and large numbers of prototypes were created, but the technology was not yet economically viable for mass scale deployment – despite the evident need for immediate deployment.

2000 – 2020: consolidation and galvanisation

Throughout this period, there was continued rapid development across all areas of science and technology. Staggering advances in medicine (from the human genome project to sequencing for $1000 a pop) to electronics, to the nascent precursor to full nanotechnology known as reticular chemistry, to artificial intelligence, and also massive development of cultural artifacts and democratisation of the means for production through 3D prototyping software, circuit design and global supply chains available at the touch of a button, this period was tremendously transformative, and also tremendously disruptive. Book stores and video stores went out of business, driven by new models spearheaded by Amazon and Netflix. Newsagencies also closed down, driven by the pervasive spread of news media.

In terms of technologies to mitigate against system shocks, there was tremendous progress here, too. Markedly, there was so much progress, that the cost of deploying renewables became cheaper than all fossil fuel sources at some point in 2017 – an inflexion point was reached.

Many countries had not waited for things to become a no-brainer economically though, and had aggressively subsidised for the adoption of new technologies, taking into account / discounting for the potential risk of not taking action in the present. Insurance companies and key financial players also altered their posture in this period, as the cold hard reason of expected return on investment and discounted value trumped any form of political finangling. In this period, one by one, the key brokers of the global economy adjusted their portfolio positions and aligned themselves with the most profitable option going forward.

It is hard to say exactly when renewable technologies became a sure-fire way to rationally invest into, notwithstanding the fact that air to breathe is normally considered advantageous, but it was at some point during this period.

Also, within these years the church saw a marked resurgence in popularity – and surprisingly not mainly in the West, but rather in the East. This increase in social capital in the world – or at least, the recognition of its value and centrality in many ways to mitigating certain forms of civilisation setback risk – is a good and encouraging thing to see.

Even in the West though, the church has been becoming stronger, driven by increases in global poverty and fractures in the fabric of society between the haves and the have nots. Paradoxically, this evil manifestation of the contemporary economic machine – a machine that seems to have forgotten people in the drive for greater efficiencies and greater automation – has caused many to be left by the wayside in a growing gig economy and/or mid to long term unemployed, and hence has caused people to start to return to the church as a source of moral and spiritual guidance. I can see this trend continuing in the decades ahead.

2020 – 2040: mass scale deployment and mitigation

It seems reasonable to expect that this period should see continued advances in various areas. One might expect exploitation of space to start to become economical. This will create new opportunities. General artificial intelligence, or some precursor to it, should likely be created during this period. Reticular chemistry will mature and products of it will start to become industrialised. Quantum computers – powerful quantum computers – will become available. Room temperature superconductivity may become available. Self driving cars running on fuel cells or electricity will dominate the roads. The remaining carbon dioxide emitters in industry, power, and transport will be addressed (such as in the production of concrete and steel).

Massive deployment of renewable technologies will continue. Costs will continue to go down, and efficiencies will continue to increase. By 2040, moreover, new technologies such as nuclear fusion should be available too. However I would anticipate that by 2040 wind, solar, geothermal, biomass, tidal and hydroelectric will be generating a large chunk of the world’s power needs. I also anticipate that there will be a lot more microgeneration of electricity.

Advances in food fabrication, such as protein scaffolding techniques, will continue in this period too – potentially reducing the need to set aside large amounts of land to grow cattle, and thereby providing another avenue to offsetting emissions.

I would also hope that the church would continue to grow and people would once again embrace it as a civilisation enabling – and protecting – cultural artifact.

2040 – 2070: crisis averted

By this period, products of reticular chemistry and other forms of nanotechnology should become available to start to do things like mop up CO2 and clean up other contaminants left around from earlier periods. General artificial intelligence will be solved. The transhumanists will have the cyborg option if they want to take it. Fusion power will be solved and people will be able to start mass deploying fusion power plants.

The world will have warmed by 1.5 degrees, faster than people thought. This will have caused hundreds of billions of dollars – if not trillions (in 2020 terms) worth of damage to the world economy. There will have been tremendous loss of biodiversity globally. But people will slog through and do the work that needs to be done. The church will help and be there for those who have suffered, as will other multinational aid organisations.

Space, at least in our solar system, will be fully opened up to economic exploitation and expansion. A base will be established on Mars. Multiple bases will be established on the moon. The asteroid belt will be exploited for rare earth metals. Outposts will be created on the moons of Jupiter and Saturn. People will be starting to look more seriously towards the stars, and pondering technological ways of getting there.

By and large, by the end of this period, mopping up the damage caused since the industrial revolution – but a necessary damage, for the development of a space-faring civilisation – will be well underway. CO2 levels will plateau, and then begin to drop – slowly at first, and then more markedly, as the scrubbers do their work of reversing our 20th and 21st century impromptu terraforming experiment. We will have solved the grand challenge of the 21st century – ensuring that our life support systems remain nominal.

2070 – 2120: maturity

In this period of time, it is possible that although we may not yet become an interstellar civilisation, we may lay the groundwork in order to do so, in the very very far future (2120 – 2200). In particular, we may learn how to develop systems capable of travelling between stellar systems at extreme rapidity – even if we don’t build and deploy them during this period.

We may make overt contact – or have overt contact overtures made from, and accepted – with alien civilisations.

In terms of custodianship of the earth, in many nations, it is likely that people will rewild the land, as the civilisation starts to become a machine civilisation. This will not likely happen in this period, because it still seems a bit too early. However, there will likely be early adopters in this period, with many people decided to “go full cyborg” and “may not even choose to avatar”. In particular, by the end of this period, many citizens – though maybe only a sizeable minority – may only have an existence in some futuristic form of server farm.

2120 – 2200: far future

Okay, maybe one more future-spective.

As this period progresses, I see that these server farms (or whatever term people choose to call them) may become the new cities. The old cities will be gradually abandoned, and maybe digital copies may be created, with the wild allowed to reclaim what was once the massive sprawl of the 20th and 21st century civilisations. Major cultural artifacts will be preserved in their physical locations, of course, and it may amuse citizens to avatar and liaise in the great cities of the world. However the true civilisation will live, work and play within the machine civilisation, which will exist within “city memory” type computing structures. As such, the demands for land for agriculture and urban sprawl will not be present. People will allow the wild to reclaim much of what was taken – but not all, for, as mentioned, it will be of value to people to be able to experience an idealised form of bucolic life on “the old earth”, or LARP in an arbitrary and contrived way in various places.

By 2200 I would be disappointed if people had not developed and deployed the means to reach the nearest stars, and with rapidity. I would also anticipate people to have developed weirder technologies still (pocket universes, etc). This is why thinking about the far future is probably largely not meaningful. But by 2200 I think people will largely still be doing people-ish things. The challenges of the late 20th and the 21st century will have been forgotten though – civilisation will have moved forward, and will be preoccupied with other concerns.

Devanuary retrospective

January 31, 2020

General morale

Great! I smashed the sprint that I set for myself, and managed to complete a large number of the cards that I originally put in there.

What went well?

Looking back at the month that was, I think it went quite well from a productivity perspective.

  • I managed to almost achieve the very ambitious goal I set up for myself, getting networking working end to end.
  • I didn’t end up getting exhausted halfway through. Rest days on Wednesdays and Sundays were a very good idea.

What didn’t go so well?

Sometimes I slept in, so I didn’t have time to do much work on the project before going to work. However, that is probably understandable, and I was able to make up the lost time by swapping out for a rest day on one or two occasions.

Questions for myself

  • In terms of in-game level editing, do I want to just use something along the lines of an editable gridmap, or do I want to go straight for something more sophisticated, like Voronoi diagrams inside Voronoi diagrams?
    • I think I should probably architect in the view that I should be allowing for both. However the data model that I use for the world state should be sufficiently flexible that it can allow for both.
  • Is there a risk that the dndsandboxserver might become opinionated in terms of the logic in the dndsandboxclient? How can I protect myself from this sort of thing?
    • This is a risk. Ideally we want to make sure that the dndsandboxserver needs to be sufficiently abstract that it is not dependent on any logic in the dndsandboxclient. If I do this, then at some point down the line I could potentially sell it as a service for people to use.
    • Ideally I should create technical debt cards if I feel that the client logic is leaching into the server logic in any way, shape or form, and then prioritise these cards accordingly in the backlog of things to do.
  • Should I be writing tests for the client and the server?
    • Short answer: probably I should, but due to time constraints, I’m not going to prioritise this work (yes, I know, this is a lousy answer – but at least it is an honest one).
    • Longer answer: in the long run I should probably add unit tests as part of polish before or after shipping an EAP – assuming that that is my objective. Writing unit tests for Godot games is not something I’ve seen anyone do either, so it will require some thought as to how to do. Rails unit tests are of course fairly easy to write – that pattern is well established.
      • Really, it is a matter of the amount of time I have for this work. I would like to be able to get some form of MVP ready first before thinking about unit testing things and maintaining quality of the product. I don’t have customers yet, and certainly I don’t have customers with expectations based on doing business with me for some time, so things like this are not so critical. However, if this is going to eventually be a paid product, I should have something in place to test things – i.e., have documentation as code.
      • This is first and foremost a hobby project that I am doing on my own time, and therefore I am more interested in exploring what I can build in a slightly less disciplined way first, than focusing on resolving technical debt and observing good testing practices from the get-go.
  • What do I foresee as obstacles to shipping a product?
    • I still need to make my apis authenticated. This is an issue. Another is that if I want to make my service a paid product, then I would need to find a way to integrate with a third party credit card transaction handler, like Stripe (I certainly don’t want to be handling credit card information myself, no no no …).
    • I also need to make a long term decision about whether I want to stay on rails. For the first customers maybe it doesn’t matter, and maybe it doesn’t even matter at reasonable scale, but it is worth thinking about.
  • How far to product do I think I am?
    • Well, that is a tricky question to answer! And certainly a loaded question. Considering that I’ve been talking about this project literally for years, it is risky to signpost anything. However, if the rate of progression in this month is maintainable, and I do that once every two months … then:
      • by end March I should have a good idea as to how to handle in game level editing
      • by end May I should have managed to get the rubber hitting the road on that
      • by end July I should essentially have that done – and be onto avatars in the game.
      • by end September I should mostly have the avatar tokens done
      • by end November I should have tokens implemented and be onto permission systems
      • by end January 2021 I should have permission systems done, and be moving on to polishing things up including authentication
      • by end March 2021 I should be halfway through implementing authentication
      • by end May 2021 I should have finished authenticated APIs
      • by end July 2021 I should largely have finished polish on the dndsandboxclient and have that in EAP
      • by end September 2021 I should have managed to integrate stripe with the dndsandboxserver
      • by end November 2021 I should be in a position where I almost have a shippable product,
      • and then, by end January 2022, I should (maybe) be in a position where I can launch the dndsandboxclient as an example of a game on the dndsandboxserver platform
    • So I guess that means I’m two years to product, provided that I continue to steadily plug away at this project! That is a bit disappointing, but I guess at least it is a moderately realistic estimate. Of course, if I manage to worth three months in four, the above estimate could be compressed to 16 months; less than one and a half years from now. Of course, that might not be a sustainable way to work, so 2 years from today, riding on the assumption that I remain moderately productive and focused on this project, is probably a more realistic estimate.
  • Do I have the resources I need for in-game level editing? Should I do some more legwork first before taking on that complexity?
    • I think ongoing research is important. Certainly any code under the correct license that would help me achieve my objectives, or any courses / materials that are free and/or moderately cheap would be a reasonable investment and I should definitely consider getting.
  • Are there any courses that I should take to supplement and / or help enable me to succeed in this project?
    • Yes, well there is certainly GodotGetaway by Canopy Games. That is an excellent course. I also have access to the 1-bit Godot course by Heartbeast, and a few GDQuest courses as well – which I have not yet started due to competing demands on my time. I should work through all of these really if I am serious about “developing my Godot-engine muscles”.
  • Yes, but what about other things other than just courses for Godot?
    • Ideally, yes, I would like to find something decent to read and / or work from examples in relation to hardcore procedural generation – and also how to represent the data from same. Some more leg work and/or ongoing leg work is probably required here.
    • Potentially I would like to stagger months like January “hack months” with “learning months”, where I focus almost exclusively on things like the above. Certainly in January I still did some work on auditing GodotGetaway lectures, so there is no reason that something should be exclusively a learning month or a hack month … probably rather just more of an intentional focus one way or the other.
  • Do I need a break before jumping back into the project again?
    • I think so. I’m certainly not planning to work on this for the first half of February. Certainly by March though I’d like to get back into things again, and then take another break in April.

Actions

  • In terms of a two month time horizon:
    • February shall be a rest month / learning month. I’d certainly leave the door open to work on the project in this month, but I’m not going to hold myself to it.
    • March will be a working month.
  • Wednesdays and Sundays worked great in terms of days to rest. I will continue with that going forward.
  • Using Jira was very useful. I will continue with that going forward, and, on any day that I am working, make sure that I review the sprint board first.

Devanuary 31/1: limited progress with handshaking

January 31, 2020

I managed to:

  • Get refresh servers working
  • Format columns correctly
  • Have an ‘on select’ option for a joinable game
  • Was able to send through a call group to join a server with session owner ip and port

However, I was unable to fully instantiate the new player within the running session. I might have another attempt at that tomorrow.

Devanuary 30/1: presenting data within the client

January 30, 2020

Today I:

  • added clients to the servers payload. In the end I decided that the client can decide which clients are inactive based on their last updated_at timestamp; too many requests to the server from clients would be inefficient. Client machines should have sufficient compute for this.
  • managed to display the servername, client count, and can join status for each active session. I set a flag for can join status based on whether clients were < 4 for a particular session (so that one would cap players to 4 for each session).

Still to do:

  • Format columns and data in the itemlist properly. Maybe I could dispense with the labels I created and just add a header row for my itemlist that can’t be selected with the column names I want. I can probably programmatically change the formatting in code of that single row, too – and also prevent players from being able to select it.
  • Add a timer node to the join game room popup scene, so that every 5 to 10 seconds or so another fetch request is made to the servers.json endpoint, and then the server list updated. This would be to address a problem which is that if a client A wants to join a session, and none exist, but then another client B creates a session, that session is not displayed in A’s view. This would solve this problem.
  • If A and B are looking to join a game, and C has a game with 3 players already, find a way to prevent A and B joining concurrently and blowing over the player limit. Maybe this could occur as a consequence of the next step – during the peer to peer connection handshake, if the session host has client limit number of players already (eg 4) then the handshake could be declined and the client returned to the main lobby screen. This would be difficult to figure out though because calls will be asynchronous. There should be a way of implementing a safe ‘unwinding’ of a handshake here though.
  • Implement on_listItem_selected to establish a peer to peer connection with the ip of the session host A of a particular server if B clicks on same from the join room screen. B should then join the game of the session host A.
  • Finally, find a way to resolve the bug where if one clicks Host Game without having an ip resolved, then the game crashes. This should be easy to fix.

Devanuary 28/1: join game room mockup started

January 27, 2020

Today I mocked up a Join Game room popup node in my Lobby scene, and succeeded in fetching a list of servers from the herokuapp while another game was running, and just returning one server.

Still to do is to figure out how to present this data in an itemlist in the popup, and how to wire things up so that a player can actually indicate they want to handshake with the session owner of a particular session.

Also to figure out is whether I want to return all clients with the list of servers (I probably do) and then whether I want to figure out on the client end before presenting this data in the UI as to which and how many clients are still active (I probably do want to do this on the client).

Finally, I would like to be able to send a ‘refresh’ signal to the herokuapp to refetch sessions every so often while the join game screen is active.

Devanuary 27/1: showing only active servers

January 27, 2020

Today I managed to get the herokuapp server to show only the active servers, and /servers/:server_id/clients.json to show only the active clients for that server. I didn’t use active job to do this; instead I set a datetime column in the client and server tables, and then updated this column when an ‘update’ heartbeat was sent from the client to each table individually (only the session owner can send a heartbeat to the server per my logic).

Next, I would like to fetch from the relevant endpoints and display that information to the client in a control node. After that, I would like to wire up that control node and allow the client to join a server.

Unrelated, I would like to only be able to host a server or join a game once the client ip has been returned from the herokuapp, so I will need to implement some way to freeze / lock these options until that information is available.

25/1 Devanuary: slow going

January 24, 2020

Today I realised that I probably don’t need active jobs at all, I just need to provide recently touched servers, i.e.

# GET /servers
def index
# show servers that have only been updated recently
@servers = Server.where(updated_at: 5.seconds.ago..DateTime::Infinity.new).order(updated_at: :desc)
render json: @servers
end

However, this doesn’t quite work. In order for updated_at to change, an attribute of the model needs to actually change … so sending a heartbeat won’t cut it necessarily.

In particular, I believe I need a new DateTime field in lieu of ‘is_active’ on the Client and Server models, and then change that instead. After doing that, I can show things that have recently been updated.

In terms of the client code itself, I would also like to send a second heartbeat to update Server model iff the client is the session owner.

However not much progress today. Mainly trying things and not getting very far.

Devanuary 24/1: Defining the ActiveJobs we will need

January 23, 2020

Today I slept in, so not much of an opportunity to get things done today. However I did think a bit about the active jobs I want to create.

Since the client is sending a heartbeat to the server already, each client has ‘updated_at’ set within the system. Therefore, we probably don’t need to mark clients themselves as inactive or active, since the system knows this already – time.now minus updated_at > THRESHOLD_TIME_INCREMENT should provide a good rubric as to whether a client is no longer connected.

However, we would still like to know if servers are active, since ultimately we’d like to expose those to clients who are looking for active servers!

In terms of doing this, a client could query an endpoint on the rails api server, and then this would fetch Servers which are active. How do we know though if a server is active? Well, we could have an ActiveJob that

  • runs only for active servers every minute or so, and:
  • for each of these servers, checks for active clients. If clients are zero, the server is inactive. Mark server as inactive.
  • if there are clients, for each client, call the “clients_cleanup” job. This job 1) checks if time.now minus updated_at > THRESHOLD_TIME_INCREMENT. 2) If so, marks this client as inactive.
  • if there are no more clients left on the server that are active after the previous step, mark server as inactive.

Then, after the outer job has finished running, we have a list of active servers with a count of clients that are connected, along with the ips of the clients that are connected, and whether or not these clients are the session_owner. Return this as a json response to the client.

The client could then present this json response in a Control node i.e. within the UI. The user could then have the option to click on an existing server, and the client should then be able to form a peer to peer connection with the ip of the session owner in order to join the running game.