Grooming the backlog, planning, and starting October sprint

Today I groomed the backlog a bit for the sandbox project, closed the last sprint, and started a new one.

Previous Goals

The goals of the September sprint were:

• Complete authentication implementation for Sandbox project
• Harden the Communications Subsystem
• Invoke ProtonGraph from Godot.

I made some headway on the first objective, but it turned out being a bit bigger than I originally anticipated. I did not really do any significant work on the other two items.

New goals

The goals for the October sprint for the sandbox project will be:

• Improve the Instance join experience
• Invoke Protongraph successfully

I will just roll over the September sprint into the October one, as I had already brought in a number of new cards as part of the emergent complexity discovered in the previous sprint.

September retrospective

General morale

Moderately relaxed. As I write this on the 20th of July 2021, my city is within yet another lockdown as of last Friday. Also, for the last few weeks, NSW has been in lockdown. Nonetheless, I am feeling moderately relaxed, as it seems that things are moderately under control here, and, although things are not yet totally under control in NSW, the optimist in me suspects that things might be starting to turn the corner in that respect.

I have been reading a bit more and slowing down a fair bit in terms of time spent on the hobby project. Nonetheless I’ve still been doing a little bit of work on it off and on as the weeks have gone by.

Since last month’s retrospective and now, I have been able to make a short trip to Gippsland, where I was lucky enough to be blessed with some good weather – ideal for walking. Wilson’s Promontory, Agnes Falls, and Fish Creek were amongst the places I visited. It was a good break.

And just before the current lockdown, I managed to get away again, for a short break in the Dandenongs near Monbulk. It was very pleasant there, and once again I was blessed with a span of good weather. I visited the Silvan Reservoir and also Ship Rock. I saw the Puffing Billy terminus. I purchased a tasty (and delicious) cheesecake. Things were nice and pleasant.

Shortly after my trip to Gippsland, I read David Attenborough’s witness statement, which was actually relatively uplifting, as there were a number of solutions presented therein that I found helpful and thought provoking in terms of thinking about the associated issues. I posted a book review here summarising my thoughts on it.

I also started collaborating on a short story with a friend of mine – more on that hopefully at some point in the future.

I finished reading The Evening and The Morning by Ken Follett, which was a good light-hearted bit of historical fiction, and have just started re-reading the Belgariad by David Eddings.

… and I started reading A Better Way to Live, by Graham Hooper (which is a commentary on two of the books of wisdom literature, being Psalms and Proverbs, in the Bible). Book review hopefully coming soon. There have been a number of book recommendations I’ve come across recently. From the radio, both BBC and RN Radio: Novacene, Uncanny Valley, The Screwtape Letters, Conflicted, The Constitution of Knowledge. Not sure how many of these I might read, but I will come to a decision regarding these eventually. “How to Talk about Jesus” by Sam Chan is a book that is recommended by my church. I might read that after “Age of EM” (Robin Hanson) and “Novacene” (James Lovelock).

In terms of books I continue to eagerly anticipate: The Doors of Stone by Patrick Rothfuss (and I have heard intimations that there might yet be a fourth book in that series) and The Winds of Winter by George R.R. Martin.

I’ve continued off and on with the calisthenics, and have been trying to go for a short run once every few days. Walking has also been an essential part of my routine.

In the house, there has been some simplification of possessions and better use of available space. I have aspirations in particular to get rid of some things from the shed.

I am still on Sabbatical, but this ends soon (as of the end of this week). I have been starting to prepare for returning to work. I logged onto company systems yesterday and quietly started setting up a new machine and also figuring out where things were at; I plan to continue doing this throughout the coming week. There is also a project that might be worth working on for the office too, over the coming days, as I broke this prototype system just before I went on leave. I won’t indicate exactly what this system is, however, as 1) it is commercial-in-confidence, 2) I intend to firmly keep paid work and sandbox project work separate, and 3) on this blog I will only be discussing my hobby project work. Nonetheless, it should hopefully provide me with ways to build better functionality for my team and my group (if successful) – and also potentially lead to better and more interesting work for myself at the company going forward (again, if successful) – so therefore it seems relatively sensible to spend at least a bit of time this week working on it.

I think this Sabbatical has been successful. I was able to resolve the uncertainty regarding house moving (I am staying put), as well as resolve some uncertainty regarding my sense of what I want to do for the next span of time (more of the same). I am feeling more relaxed. I have made significant strides on my hobby project and fleshed out the direction there in moderately decisive brushstrokes. I had time to get away not once, but twice. I started work on a short story (which could potentially become a medley of short stories), I did some exercise, and I did some reading, as well as spent some time in rest and contemplation. I am ready to return to work.

The Pandemic/Vaccine Situation

According to our world in data, Australia is now sitting at 39.48 vaccine jabs per 100 people, which is a significant improvement on my update in terms of the “August*” situation (*JIRA time). Note that the vaccination rate – full vaccination rate – should ideally be at least 60 to 70% to rationalise doing away with severe lockdowns, and we have a fair way to go in that respect. In other words, we need a situation similar to that in the UK, where the vaccination rate is currently close to 70% in people over the age of 18. Even then, it is debatable whether all control measures should be completely dropped, with a continuing debate ongoing at present in the UK regarding this – the Netherlands, apparently, is a bit of a cautionary tale in that regard. Australia’s full vaccination rate is only a touch over 10% at present (at 11%), which is nowhere near high enough for that sort of discussion to take place. This has not stopped senior politicians from starting to voice opinions in this regard, however, probably affected and influenced by proximal information distance bias (in terms of culture, language, ethics, trade, social networks, and religion, amongst other things) between Australia, the US, and the UK.

However, it is probably about the right time to start considering strategy, at the very least, regarding vaccination certificates and what that would look like, in terms of incentivising people to get vaccinated and thereby improve these figures, once, with any luck, we finally have the supply issues sorted out over the next few months.

I have little appetite to discuss the shambolic rollout and procurement issues, but, suffice to say, things came to a head between my last update and this one, with two of Australia’s largest states now in lockdown.

As mentioned in my last update, my appointment for my first dose will be on the 22nd of July, in two days time. So, with any luck, by the time I write my next retrospective, I will finally be fully vaccinated.

A day or two ago 1 million doses of Pfizer vaccine arrived in Australia from Europe (up from 300 thousand per week in previous weeks), as, for whatever reason, the delivery schedule was brought forward, for which I am very grateful. I gather that this accelerated delivery of vaccine will likely last for another few weeks at least until mid-August.

Vaccine from the US is not forthcoming as yet. I believe that Australians will likely need to wait until September or October for the Moderna vaccines to arrive. I believe that this is supposed to be part of our booster strategy (do we currently have a coherent booster strategy?), but I guess things are a bit of a wait-and-see here.

In terms of building domestic mRNA vaccine production capability, word on the street is that things should be hopefully up and running domestically by 2023. Again, this will likely be a wait-and-see. But actions are being taken now, and conversations have been happening with various companies (including Pfizer and Moderna). So hopefully within 800 days domestic production capability will be online.

What went well?

A lot of the work this month was largely invisible and incremental, but nonetheless I made some headway on a few things:

• Storing instances by key rather than name in Redis;
• Updating the logic on the client and the instance service to support this change;
• Ideating what microservices could look like for the communication subsystem;
• Made progress on figuring out what the architecture for the motivic subsystem could look like, and how it would work;
• Replaced some legacy logic in the client with a second WebSocket connection (to fetch instances), thereby laying the groundwork for what could be required to realise microservices for the communication subsystem.

It definitely feels like I am currently in “grind mode” for the project. But I remain at present upbeat and happy to Embrace the Grind, as the work I do now on these matters will pay dividends later.

What didn’t go so well?

I broke sprint discipline per this post. A bit unavoidable, though, I think.

What’s the outlook?

Good. A fair bit of grind remains, but hopefully (hopefully!) I will be able to find time to put Protongraph through its paces over the coming month.

Can join by key of instance

Today I:

• Fixed things so that one can join by the key of the instance again (it turns out that I was trying to join on the key of the peer, and not the key of the instance – a simple enough fix);
• Did some tidying up in the communication subsystem;
• Fixed a bug wherein a try catch was not being used to remove a key written to Redis when the write to the Instance service failed;
• Removed the requirement that instance names be unique from the instance service, as we now are using ULIDs to identify them.

Next:

• I noticed that instances with no Peers are now listed as potential instances to Join in the instance list. Although this is not necessarily broken functionality, I would like this to only occur
  • 1) for Organisations that are not the Anonymous organisation — otherwise all such instances should be hidden (i.e., essentially ephemeral),
  • 2) If an Instance has no Peers in it, on Join the new peer should become the host of the Instance and be able to enter the Instance. Currently this does not work – one cannot join an instance that has no peer-as-host.
  • 3) The number of players connected to the instance should be displayed in a column, which indicates that this information might need to be written to the Instance service on peer join and exit,
  • 4) It should be possible to delete an Instance if you are the Organisation Owner, or the user that created the Instance.
• The Instance metadata displayed in the UI should have more information, such as a short description potentially as to what they are about, or at least the Handle of the player that created it if they were logged in, or the datetime that it was created.

Fixed minor ULID bug + started working on Join

Today I:

• Fixed the ULID bug for anonymous instances;
• Wired up the button in the client so as to send a signal to the communication subsystem to join an instance;
• Ensured that the button has-as-name the ULID of the instance it wants to join, so that it can construct the aforementioned request properly.

I haven’t quite got join working, however, as the communication subsystem is complaining that “the instance does not exist”. Curious about this … according to Redis, certainly at least the key for the instance does. This will require more investigation.

Rewired request for instance list

Today I:

• Created a second WebSocket connection in the client, and wired up all necessary signals to get this working, as well as generalised all other functions and odds and ends required to keep existing functionality working;
• Removed the direct API request logic from the client to the instance service and replaced this with a request via this new WebSocket to the communication subsystem, and directed the incoming request in the communication subsystem to the instance service;
• Processed the return response from the instance service, onforwarded said response from the communication subsystem back to the client and rendered it therein;
• Made dev_mode default in the client (for now) and tidied up the UI a bit more.

I also fixed a couple of minor bugs along the way.

One thing that I noticed was that if a user is not logged in, then instances (anonymous ones) are not created with a ULID. I will fix this next, along with ensuring that I name the buttons in my instance list according to the ULIDs returned in the instances packet. Then, when I fire off the “Join” request initiated via a signal from the button “on pressed”, I intend to read the name of the button node (the ULID) and use this as the requisite data to join the appropriate instance for the prospective joinee.

I’d also like to fix the instances-with-passwords regression, I should file a bug for that.

Aframe & three.js have come a long way

I've always said #VARTISTE wouldn't be a 3d modeller, since there are so many for VR already. But sometimes you just need a cube and a sphere. So I've added shapes you can build into a drawable mesh or reference object!#VRArt #VR3D https://t.co/FaYM1vR0JU pic.twitter.com/9DJHhLizQz

— Zach Capalbo (@zach_geek) July 17, 2021

Presenting instances in the client

Today I:

• presented the active instance list a little bit better in the client
• ensured that I retrieved from the instance service only those instances per the organisationId in the filter parameter of my query
• fixed a bug with the communication subsystem
• fixed an issue with the instance service where it did not return values in local development due to the default value for config.force_ssl
• tidied up some other issues in the client associated to presenting the instance information returned from the instance service

Todo:

• wire this up indirectly via the communication subsystem instead using signals, so that I can remove the messy API implementation in the client as at present, and improve the performance of this functionality a tad
• figure out if I need to present any more information from the instance service and thereby need more columns
• ensure that the ULID for the instance is bubbled up to the client when fetching instances
• rename the button so that it has the name of the ULID associated to the instance
• wire up the join button to the appropriate function so that it passes the ULID for the instance to the communication subsystem on click
• fix a regression with instances-with-passwords that I believe that I have introduced by deleting a few blocks of code in the client

Also, a bit of housekeeping:

• Set dev mode as a Globals variable, so that I don’t have toggles in multiple places set “off” when dev mode is actually on
• Maybe default dev mode to “true” (for now) to simplify my workflow a bit!

Ultimately I’d like to use some form of playerprefs as well; it would be handy to be able to use profiles so that I can run the client with a profile, and thereby be automatically logged in as so-and-so without needing to go through the song and dance of clicking “Login”, typing in an email address, typing in the password, and clicking “Submit”. Eg ./sandbox_client –profile=myFavouriteProfile1 and ./sandbox_client –profile=myFavouriteProfile2 should launch the mainmenu scene of the client while already logged in as so-and-so of such-and-such organisation. But of course this would also likely necessitate changes on the backend services, too. And also, this would likely require using something like SQLite for the client, and thereby be quite a large piece of work. I’ll create a card for this, however, as this is something that I think would be useful to have for other reasons, too; essentially this would be the equivalent of having different “cookie & localStorage profiles” for a web browser, so potentially quite useful.

Things are a little bit garish now, but at least this is better than what I had before. GridContainer seems to be better than ItemList in many respects.

Several odds and ends

Today I:

• Ensured that the instance service has a scope to filter the instances endpoint by organisationId, and applied this to the index action on the main controller to optionally filter the displayed instances accordingly.
• Fixed an issue wherein local development was not fetching the instances from the instance service properly.
• Fixed an issue with the loggedInUserKey when creating instances anonymously.
• Fixed an issue with the communication subsystem when creating instances anonymously.

Next I intend to:

• Pass organisationId as a filter parameter from the client to the communication subsystem.
• Onforward this parameter to the instances endpoint on the instance service.
• Process the returned information from the instance service and return to the client.
• Ensure that the client properly renders the filtered response.

After that I intend to:

• Ensure that I can select a displayed instance within the client and join it as a new user.

Presenting organisation instances properly

In this post I identified that there are a number of things that need doing in order to be able to properly use ULIDs, and then move to a situation where I make better use of Redis rather than having a couple of keys with very large objects attached to them.

To action this, then, I’d be after fixing presentation of instances in the client. This will require:

• creating an endpoint to return all instances that belong to the active organisation on the organisation service
• hitting this endpoint from the communication subsystem
• this communication subsystem call should be invoked from the client
• the response from the communication subsystem should then be rendered into an itemlist, with invisible ULID data associated to each item
• when a user selects an item in the itemlist they should be then able to join the instance by ULID
• the ULID then is attached to the “Join” call and everything works.

Also, I’d be after allowing owner users to add standard users to an organisation. This will require:

• Providing an owner-specific settings panel in the client — this will also require changes on login to attach to a user the information that they are the organisation owner (or not)
• In the owner-specific settings panel a user should have access to a form to add a new user
• When the new user form is submitted by the organisation owner, this request should be sent to the communication subsystem, and onforwarded to the organisation service (and/or user service) wherein a new endpoint has been implemented to create the new user. (This may require a transaction and appropriate use of consumers and producers re: the kafka VM so that both services can be interacted with properly.)
• Upon successful creation of the new user, a receipt should be bubbled back up to the client so that the organisation owner knows that their request was successful.
• Upon successful creation of the new user, an email should be sent to the new user’s email address letting them know that an account has been created for them with a particular password. They should then have a confirmation link to confirm their account, wherein they will be further prompted to change their password should they wish to do so.

Other considerations:

• All users in an organisation should have access to settings wherein they can change their password. This means that I’d need to wire up a password update endpoint on the user service to the client with the communication subsystem as intermediary.
• Per this blog post (Building a fault-tolerant event-driven architecture with Google Cloud, Pulumi and Debezium), I’d be after using something like Debezium to implement a write ahead log so that I can ensure that I understand whether or not all changes have been applied or not in a particular flow. Here is another blog post about using Debezium to solve the challenge of building fault tolerant systems.

So this is a reasonable amount of work. Likely I can defer looking into Debezium (and maybe also replacing my Terraform code with Pulumi), but I should probably look into actioning the other items before seeking to move to a one-key one-instance model with Redis.

In view of the above, I’ve drafted another small batch of cards and plonked them in the current sprint, as they block the work on moving Redis to the one-key one-instance model. 18 more points! Hahaha. Looks like I’m going to be rolling over the sprint again next month and not bringing anything new in when I am actually supposed to. Oh well … I guess I’ll eventually finish the current batch of cards.

“Talk to a Scientist” – outcome

Perhaps unsurprisingly, the Talk to a Scientist investigation was a flop.

My email:

Dear admin,

I write as I am interested in setting up a time to sound out the suitability of your organisation for soliciting feedback on a research programme that I’ve worked on over the last 15 years. You can find a summary of it here.

Should such suit, I’d be after a 1 hour consultation to get started.  Please note that my local time is UTC+10 (AEST).

I look forward to hearing from you.

Best Regards,

<First Name>

Their reply:

<First Name>,

Unfortunately, we do not offer this kind of consultation service.

-<Initials>

So it appears that the representations made herein by the relevant third party were not correct and a trifle misleading – or, more likely, said researcher/blogger simply misunderstood the nature of the Talk to a Scientist service.

I might therefore sit on things here for a bit and see whether God wishes to open up and/or direct me to any other avenues with regards to this particular knowledge/research ministry. Perhaps following up with the Oxford Quantum Foundations organisers might be the way to go. But I have no desire for a direct approach. My desire to reach out assertively to seek feedback on my research has ebbed for the time being.

Trying to learn a bit more about the Oxford Quantum Foundations group (who reached out to me indirectly a number of years ago) and seeing if I can rebuild some bridges there seems like a better approach to make some oblique inroads towards obtaining feedback on my work. Therefore, I will follow up with one of the members of said establishment (from whom I last have a communication from back in 2011) and see if I can get back on their mailing list; I will action that today.