Archive for May, 2022

Grooming the backlog, planning, and starting June sprint

May 31, 2022

Today I groomed the backlog a bit for the sandbox project, closed the last sprint, and started a new one.

Previous Goals

The goals of the May sprint were:

  • Get the Protongraph worker producing to the Kafka VM
  • Make a start on producing to the Kafka VM from the communication subsystem

I finished both of these, and also managed to consume the return packet in the communication subsystem. Achieving both goals, a first!

New goals

The goal for the June sprint for the sandbox project will be:

  • Get the end to end flow for the creative subsystem working

May retrospective

May 30, 2022

General morale

A bit tired but feeling generally hopeful. As I write this on the 15th of March 2022, global events have taken a new turn, and although finding peace seems elusive, light continues to be present.

Definitely feeling better than last month. I had a chance to get away and see some beautiful natural things, which was quite pleasant. I have a renewed resolve to find places of peace and rest in my weekly rhythm.

What went well?

This month I:

  • Managed to get the Protongraph Worker producing to Kafka, and contributed to the main Protongraph repo,
  • Produced to Kafka from the comms-api service,
  • Fixed some DevOps costing issues and made some headway with rebuilding the soap site on a t4g.medium,
  • Created a new comms-consumer service and succeeded in onforwarding messages to the comms-api.

So plenty of progress this month with regards to the Sandbox project. On top of last month, yet another one or two milestones of moderate complexity were passed. I even brought more work into the sprint as I went! In particular I feel more or less at the point where crafting an end to end flow for the creative subsystem is within reach.

I also spent some time resting and focusing on marshaling my strength. Still managed to get a bit exhausted but my mental health improved.

What didn’t go so well?

I feel like I didn’t have too much time to rest this month. My intent going forward is to try to observe a Sabbath day once a week; that should help.

What’s the outlook?

Good.

My basic plan is:

  • Alter the protongraph-worker so that it builds as a docker image and produces to Kafka on receipt of a message from the protongraph-provider. This will require altering the default make command, as well as ensuring that I process the incoming message from the provider correctly in the right spot. Possibly might be worth writing one or two helper functions (eg, a “start” script), as well as writing one or two helpful pieces of documentation. Once I’ve done that, I should consider contributing back again to the main protograph project.
  • Handle the message from the comms-consumer in the comms-api server and route it to each and every client in the relevant instanceId (per packet) for rendering.
  • Handle the packet for rendering within the client, and, well, render it.

If I have further capacity after that, I should look into persistence of procedurally generated data within the instance service in the bookkeeping subsystem. I should also decouple the bookkeeping subsystem from comms, by ensuring that Kafka sits between the two. Furthermore I should ensure that each bookkeeping subsystem service has only one correspondingly named consumer, which, if need be, consumes from multiple topics. Evidently all of this will require work and some refactoring.

For reference, here is the general picture as to how I wish to use the creative system again. I’ve now done (1) through (5), sort of done (6), and implemented (7). Still to-do is to implement (6) properly, complete steps (8) and (9), and then finally ensure that the instanceId information propagates its way all the way through so as to facilitate this flow.

I’ve decided that I don’t want to create a Delegator service, as building for Linux and running as a Dockerised service should sort out the limitations I had before with building Protongraph with librdkafka therein. Also building as a Dockerised service is good for another reason, as eventually I’m guessing I will likely want to pursue that path for all of the services in the sandbox project, for purposes of standardisation and simplifying certain DevOps considerations.

New Life + some scribblings

May 29, 2022

Recently I thought a bit about how to place a multiplicative structure on any differentiable manifold. Evidently some manifolds lend themselves very naturally to this – these manifolds are known as Lie Groups. But I am interested in providing any differentiable manifold with one or many general multiplicative structures, since I feel that some of the PDEs that I’ve been defining on standard manifolds are too high order to admit numerical solution in realtime after a practicable fashion – even with powerful topological quantum computers on hand.

My view is that if one instead pushes the structural complexity away from the PDE, and rather pushes it into the structure atop which the PDE rests, that is likely a more natural approach.

Created the comms-consumer service

May 28, 2022

Today I:

  • Copied the protongraph-provider code and renamed things,
  • Altered the topic from CommsToCreativeTopic to CreativeToCommsTopic,
  • Altered the target websocket port from the Protongraph-Worker port to the Comms-Api port,
  • Built and tested the project successfully, through obtaining a message emitted from the Protongraph-Worker indirectly via Kafka, and onforwarding it to the Comms-Api service, wherein it was printed to the console output.

Next, I should seek to wire up the Protongraph-Worker properly to handle a message from the Protongraph-Provider, as currently I’m just hardcoding a message for it.

Architectural vision

May 27, 2022

This is currently the vague plan. At present the connection from client to coturn and to comms-api is direct, but I’d like to place a Gateway in-between so as to secure these services.

The communication between comms-api and instance, user and org is currently direct. This should be changed to be via Kafka instead. The corresponding consumers ic, oc and uc can continued unchanged.

The provider exists, but should route to the delegator (which doesn’t exist yet) rather than sending packets directly to Protongraph (the Worker).

The comms consumer should route packets to the comms api.

Finally, a new service the “Bridge” should allow for clients to move from one instance to another should they so choose – think “for instance” of a door into a building, or a map transition. In this way, a client’s connection should be able to move seamlessly (in theory) between different instances within the same “campaign”. Evidently it would be good to be able to support multiple instances per client too, so this diagram is not necessarily representative of that — or rather a “client” would be a “client instance object” instead, but the above is already good enough for a POC.

Book Review: The Ruthless Elimination of Hurry

May 26, 2022

Today I finished reading “The Ruthless Elimination of Hurry” by John Mark Comer, which can be viewed as a entrypoint to approaching the commentaries of one of the more prominent theologians of the late 20th century, Dallas Willard. In particular, there is a quote attributed to a conversation between a mentor of John (also, confusingly, called John), and Dallas Willard that provides the title of this book.

John: “What do I need to do to become the me I want to be?”

Dallas: “You must ruthlessly eliminate hurry from your life.”

John: “Okay, what else?”

Dallas: “There is nothing else. Hurry is the great enemy of spiritual life in our day. You must ruthlessly eliminate hurry from your life.”

In a nutshell, the book has the following format:

  • What the issue is, and why it is an issue,
  • How one goes about solving this problem (the theory)
  • How one goes about solving this problem (the practice)

Basically, to condense down part one of the book, the problem is – you guessed it – the frenetic pace of modern life. Living life in a hurry to get from moment to moment, or to achieve one thing after another, seems to not be a satisfactory way to go about doing things – or living.

For a rough and ready definition, hurry can be defined as trying to fit too many things into one’s day, necessitating the need to rush to get all of them done.

Byproducts of hurry include fallout with one’s physical, mental, and spiritual health, as well as one’s relationships with other people – amongst other things.

So how do we solve this problem? In a nutshell, it boils down to this:

28 “Come to me, all you who are weary and burdened, and I will give you rest. 29 Take my yoke upon you and learn from me, for I am gentle and humble in heart, and you will find rest for your souls. 30 For my yoke is easy and my burden is light.”

Matthew 11:28-30, NIV

Essentially the solution is not to escape work or burden – this is unrealistic. Life will always be difficult and challenging. Rather, the solution Jesus offers is equipment, and to be yoked as a younger ox to the stronger ox, Jesus, who will help us to steer the plow.

In particular, the solution to hurry is not to stop working. Indeed the Proverbs of Solomon counsel strongly against it:

Go to the ant, you sluggard;
    consider its ways and be wise!
It has no commander,
    no overseer or ruler,
yet it stores its provisions in summer
    and gathers its food at harvest.

How long will you lie there, you sluggard?
    When will you get up from your sleep?
10 A little sleep, a little slumber,
    a little folding of the hands to rest—
11 and poverty will come on you like a thief
    and scarcity like an armed man.

Proverbs 6:6-11

On a slightly different tangent regarding approaching work:

15 “I am the true vine, and My Father is the gardener. Every branch in Me that does not bear fruit, He takes away; and every branch that bears fruit, He trims so that it may bear more fruit. You are already clean because of the word I have spoken to you. Abide in Me, and I will abide in you. The branch cannot itself produce fruit, unless it abides on the vine. Likewise, you cannot produce fruit unless you abide in Me. “I am the vine; you are the branches. The one who abides in Me, and I in him, bears much fruit; for apart from Me, you can do nothing.

John 15:1-5, NIV

A useful image that is provided in the book is that of the trellis that bears the vine; without this, it is not possible for a vine to grow and produce fruit. So it is with various practices associated to Christianity. The author of the book also speaks about abiding – in scripture, in prayer, and other spiritual disciplines. That forms the third part of the book.

In the third part, broadly four main disciplines for relinquishing hurry are provided:

Silence and Solitude. This is the seeking out of lonely places (“eremos”) for seeking God and resetting. In more secular approaches, this is what is known as “mindfulness”. But in the Christian practice, there is more to this than merely emptying one’s mind of clutter and noise – one does that, yes, but one doesn’t fill it with void. Instead, the idea is to fill it with scripture and meditation on what God might be speaking into one’s life.

Another key idea introduced here is the concept of working from a place of rest, rather than working in order to obtain rest.

Sabbath. Sabbath is about setting aside a day each week not as a holiday, but a day to focus on rest and worship of God. Interestingly, apparently in ancient Israel there were three harvest festivals of 8 days, making for several longer sabbaths to punctuate the year – which is in keeping with contemporary findings that 8 days is about the right amount of time to slough off the stress of working. Another interesting observation is that 30% of the wording of the ten commandments was actually given to asking the Israelites to honour this day and “keep it holy”. The focus there is on rest and worship.

The Sabbath is also mentioned in Deuteronomy, where Moses alters the emphasis, focusing on a different aspect to it – of resistance against the system of the world. In particular he reminds the Israelites just before they are about to enter the promised land that their parents and grandparents had been slaves, oppressed under the Egyptians. In particular, slaves don’t get breaks, they don’t get holidays. They are chattel, assets to be used to build up the coffers of their oppressors, to be ultimately broken and ground into dust under the weight of their toil. In particular, Moses implores the Israelites to observe the Sabbath so that they will not fall back into slavery. An intriguing observation is that the author observes that “Egypt is all around us today” [in the form of the drive to hurry, to accumulate, and to seek to either serve as slave or slave driver in pursuit of a nebulous more]. And maybe it is true! So keeping the Sabbath is a serious business!

Simplicity. Simplicity is about seeking not to accumulate more, but being content with enough. “Life does not consist of an abundance of possessions” (Luke 12:15).

Slowing. Slowing is not a traditional discipline but is rather a contemporary response of “followers of the way” to the great acceleration that we are seeing around us, as our Techne continues to progress at a manic rate. Things such as “parenting your phone”, i.e. putting your mobile phone to bed early (8:30pm) and waking it up late (not checking it until 7:30 or 8 in the morning), not constantly checking your email (by eg scheduling certain times to check it), and even taking it off your phone entirely, deliberately choosing the slow lane on the road or the longer queue at the supermarket, turning your smartphone into a dumbphone by removing internet, notifications etc.

Another suggestion regarding slowing that resonated but nonetheless seemed challenging was the idea of having a periodic retreat (once a month – that seems way too frequent!) to take stock of where one is at and meditate and reflect.

Others such as little or no social media and little to no TV seem also quite sensible (I agree with the author that there is very little on Netflix that is worth watching anyway!). Single-tasking rather than multi-tasking also seems like a great suggestion – I’ve always railed against being told that I need to “multi-task”, since I prefer to focus carefully on one thing at a time. It is good to learn that this at least forms part of the author’s philosophy regarding slowing.

Regardless, after reading this book and paddling in the shallow end, perhaps I might be ready to read some theological commentaries and/or philosophy by Dallas Willard – I should try to seek out his work at some point.

Wired up messages from client to Creative

May 25, 2022

Today I wired up sending a message from the client all the way through to the creative subsystem. After testing things end to end, I was able to get the packet routed to the Protongraph-Worker, but not back to Kafka from there … I will need to look briefly into that before progressing a bit further.

Regardless, after that, which hopefully should not take too long, I’d like to focus on dealing with the return message from the Creative subsystem, i.e. I’d like to start drafting a consumer based loosely on the Protongraph-Provider service, but on the other side of the fence.

Various changes to client

May 24, 2022

Today I:

  • Added sync-godot, concept_graph boilerplate to the project,
  • Added fence.tpgn, drafted a new generator type of clipboard object,
  • Modified some of the clipboard paste logic to account for different clipboard types,
  • Made some other small quality of life changes.

Next, I’d like to see if I can trigger a call to Protongraph via the communication subsystem when I paste a generator. i.e., I’ll want to avoid using the sync-godot logic entirely (which is the old network topology for local side-by-side running of the Protongraph binary) and instead send a message via Websocket. The spot to do that is the paste_clipboard_object method in EditorObject.gd, wherein I’ve put placeholder code: print(“You just placed a Generator, congratulations.”).

So good progress I think, I just need to craft a reasonable packet to send to the comms api server and then I can start checking additional bits and pieces of the round trip.

To recap, I’m basically looking at wiring up steps 1 and 2 in the below diagram:

I’ve already done 3 and 4. Step 5 needs re-doing, but I can potentially check the older network topology first.

In particular, if I go by the second diagram below, then after what I’ve looking into now, I should be able to route a request through steps 1 through to 6. I will then be able to focus on the comms-consumer logic, step 7. Evidently I’ll need to create a new app for that, but I can probably crib from the work I did on the Protongraph-Provider. C++ should be the go I think in terms of the tooling for it, because, well, it is highly performant.

Started drafting changes to client

May 23, 2022

Today I started drafting changes to the client in order to support a procgen command emitted to the comms api server.

One of the things I am looking into introducing is an abstraction to simplify emission of a packet to said system; this should reduce the number of LOC in one of the files hopefully, although it might necessitate corresponding changes to the comms api server as parsed payloads may not necessarily be properly interpreted should they have a “|” at the end — although I could always remove the last of those, perhaps I will do that tomorrow.

i.e.

# Takes an Array and returns it
# as a | delimiter separated String
func prepare(options: Array) -> String:
	return PoolStringArray(options).join("|")

Actually, I think that this doesn’t put a “|” at the end, so likely all good.

Regardless, in ws_webrtc_client there are a number of methods that would benefit from the usage of this new helper method; I should refactor the code accordingly so as to leverage it. Naturally this will alter the invocations of said methods in ws_webrtc_client as I will be passing an Array rather than a set of Strings for various different purposes. It also presents the opportunity of coalescing a host of different methods into one, as I can then just pass a command string eg “ExecuteProcgenProcedure”, “UpdateShadows”, “FetchTokens”, or “PuppetTokenRequest”, for instance, to a common / shared function. This indeed should reduce duplication and number of LOC dramatically!

I also started looking into the UI again. I’ve introduced a new tab in the interface for “generators”. The next step will be to create a Generator scene and allow it to be associated to a 2D texture in the interface — analogous to player tokens or prop tokens are already — and populate the Generator scene accordingly along the lines of my example Protongraph project here, so as to leverage ProtonNodes etc, and also presumably leverage sync-godot somehow.

(The current texture for “item_fence” evidently needs to be updated from the “spike” placeholder image)

So a few things to do:

  • Create the Generator.tscn Scene.
  • Hook up the editor logic so as presumably to offer a custom parameter config logic in the Generators box should the icon be clicked, before “confirming” and then a fence tooltip presumably being attached to the editor avatar node for placement in the Level.
  • When the user in Editor mode clicks a spot in the Level, we then would like a command to be sent to comms with
    • The peer_id and also the instance_id (if we want a client in future to be able to support multiple instances concurrently, the latter would be useful to emit).
    • The procgen algorithm configuration params.
    • The procgen algorithm name.
    • The procgen algorithm target’s translation and rotation.

Later on, we might want to introduce the idea of validations — if for instance a procgen command is given, and then a response is returned for rendering, but said response collides with props already in the scene, we might reject the render request and send a signal back to the communication subsystem to indicate that the render request has been rejected — and the communication subsystem would then evidently instruct the persistence layers (such as Redis + the bookkeeping subsystem) to make adjustments accordingly.

Connected to Kafka from comms api server

May 22, 2022

Today I successfully tested sending a message to my secured Kafka VM for a particular topic, from my comms api server.

Two things tripped me up in the process. With kafkajs 1.16, I discovered that errors were not emitted on kafka.producer().connect(), i.e. the handshake silently failed.

Downgrading to kafkajs 1.12 revealed why – I was connecting to localhost:9092 … which is obviously a nonsense as my kafka VM is elsewhere and should be accessed at mycoolkafkavm.com:9093 (although I was attempting to connect at mycoolkafkavm.com:9092).

Digging into that, I came across this comment: https://github.com/tulios/kafkajs/issues/769#issuecomment-645988876 indicating that advertised.listeners was wrong in my kafka VM config.

I looked at that and saw that I had internal SSL (ISSL) set to localhost:9092, and external SSL (OSSL) set to mycoolkafkavm.com:9093. I was puzzled for a bit and then figured it out — I needed to connect to the VM on port 9093, not 9092, in order to pick up the correct advertised.listeners configuration!

After that things worked flawlessly and I bumped back to 1.16.

The next step will be to make the request to the comms api server from the client, and check that the Kafka topic receives the message.

After that, I’ll be writing a comms consumer app! Exciting!