Dealt with some issues I was encountering with CSRF tokens,
Solved a knotty problem involving fixing @rpc calls for the Godot 4 networking logic migration,
Drafted a new project involving writing a new Android app from scratch,
Installed a prototype of the Android project on my phone.
What didn’t go so well?
Nothing much to say here; things went moderately well. I did allow myself to get distracted by the Android project, as well as surveys of a number of interesting topics, but on the whole I think that these activities were a good use of my time.
What’s the outlook?
Reasonable.
The current plan is to continue to focus on things in the following three broad arcs:
Get to code complete for the prototype (next month)
Migrate the client to 4.0.4,
Restore remaining functionality,
Ensure that limits are altered depending on the plan type.
DevOps work (next 1 to 3 months)
Create a secret service leveraging Vault, this should be queried on build / deployment of services.
Deploy comms to DigitalOcean behind a jumphost.
Deploy instances, organisation, and user services to DigitalOcean behind samesaid jumphost.
Migrate kafka behind jumphost (out of scope: migrating from single machine to cluster of three machines).
Launch (in the next 3 to 5 months)
Deploy client to itch.io, provide optional payment feature.
Ensure that “pay for use” is disabled at the get-go and the Free tier is all there is to use.
The road ahead
Similarly to January, I am looking roughly to solve for these pieces of complexity still:
Godot 4 migration (~ 2 points)
Feature parity /w 3.5 (~ 5 points)
Validating limits (~ 5 points)
I expect that by the end of the February ’24 sprint these actions should just about be done.
After that:
DevOps (~ 30 points)
Launch prework (~ 10 points)
I’d expect these to be done conservatively in about 4 sprints, which should still take me to the end of the June ’24 sprint.
Assuming that I tread water in terms of managing my time, I should be able to complete this work within another 6 months, which would take me to around the middle of 2024.
After that, I can start my post-launch roadmap / product iteration actions. Things like tests will be near the top of the list, I think. Then probably other things as well, such as migrating to Godot 4.1.3, or Godot 4.2.x if I feel that branch is production ready by the time I get around to the work.
In particular, the things that I’d be targeting for improvement would be:
Improved the robustness of my racecar consumers by increasing their timeouts (note, I’ll need to use supervisord or kubernetes in order to restart these in case of a crash still in production),
Reversed the order of instances returned via Pagy to show the most recent first,
Fixed the meshlink ready issue for the host by implementing the AnswerAck and AnswerAckCallback pattern sketched in my previous post here
I’m starting to feel quite happy about the status of the Godot 4 migration now, I feel that I’m almost ready to declare victory on it.
One last round of testing basic functionality I think, and then I’ll just ascribe any other errors to “pre-existing bugs”, and then move to close off the current task and address those as follow-up work.
Today I figured out why the server signal is not propagating. It is because the “_on_mesh_link_established_receipt” function does not appear to be firing for a joining peer, so the mesh link is not being registered as ready, which means that in code like this
func _process(delta):
var peers = multiplayer.get_peers()
var mesh_link_established_status_store = GlobalMeshLinkData.get_store()
for remote_peer_id in peers:
if mesh_link_established_status_store[remote_peer_id] == true:
GlobalEventBus.dispatch("test_editor_avatar_rpc_authority")
rpc_id(remote_peer_id, "net_update", self.position, self.rotation)
mesh_link_established_status_store[remote_peer_id] always evaluates to false for the joining peer with id remote_peer_id.
It seems that the mesh link _is_ established for the host in the joining peer, just not the joining peer in the host, i.e. we’re only registering a mesh linkage in one direction.
Sure enough, if I delete the “if mesh_link_established_status_store[remote_peer_id] == true:” line, everything works.
So why isn’t this working as it should? Good question! It seems that it is because I’m not getting an “_on_answer_received” call on the host … and why is that? Because I’m not receiving an “A” signal from comms in the host on peer join … and why is that? Do I expect that? Maybe the structure should change with the networking changes in Godot 4?
Certainly the handshake is established, if I remove the boolean check then things do work. This indicates that I don’t need a fix in comms, or in terms of any back and forth with backend signals, I just need a signaling change on the frontend. i.e., I should have some way to call GlobalEventBus.dispatch(“mesh_link_established”, remote_peer_id) for the host when the mesh link is ready.
Maybe though … the host creates an answer for the joining peer, maybe the solution is to have special handling in comms for an answer sent by the host, and then have an “AnswerCallback” signal that then sends a message back to the host, once the joining peer receives the answer? A bit curly, but generally speaking I think this would ensure the mesh link is truly ready:
Host sends answer to joining peer,
Peer (with id remote_peer_id) receives answer “A” signal,
ARC 2 is an ambitious extension of ARC that aims to address certain weak points of ARC 1. While every task in ARC 1 was unique, ARC 1 wasn’t as diverse as it ideally would be: some types of tasks (e.g., “fill the empty patch in this patterned grid”) were reused more than once and were occasionally reused across different dataset splits.
ARC 2 is intended to be larger and more diverse than ARC 1, with no specific task type used more than once. ARC 2 will consist of tasks created by a huge community and with a great diversity.
“To make deliberate progress towards more intelligent and more human-like artificial systems, we need to be following an appropriate feedback signal: we need to be able to define and evaluate intelligence in a way that enables comparisons between two systems, as well as comparisons with humans. Over the past hundred years, there has been an abundance of attempts to define and measure intelligence, across both the fields of psychology and AI. We summarize and critically assess these definitions and evaluation approaches, while making apparent the two historical conceptions of intelligence that have implicitly guided them. We note that in practice, the contemporary AI community still gravitates towards benchmarking intelligence by comparing the skill exhibited by AIs and humans at specific tasks such as board games and video games. We argue that solely measuring skill at any given task falls short of measuring intelligence, because skill is heavily modulated by prior knowledge and experience: unlimited priors or unlimited training data allow experimenters to “buy” arbitrary levels of skills for a system, in a way that masks the system’s own generalization power. We then articulate a new formal definition of intelligence based on Algorithmic Information Theory, describing intelligence as skill-acquisition efficiency and highlighting the concepts of scope, generalization difficulty, priors, and experience. Using this definition, we propose a set of guidelines for what a general AI benchmark should look like. Finally, we present a benchmark closely following these guidelines, the Abstraction and Reasoning Corpus (ARC), built upon an explicit set of priors designed to be as close as possible to innate human priors. We argue that ARC can be used to measure a human-like form of general fluid intelligence and that it enables fair general intelligence comparisons between AI systems and humans.”
And here is a Youtube interview between Lex Fridman and Francois Chollet on the project:
Pragmatics, Neuro-Symbolic AI, and vector databases
Also in the same blog post from earlier this year, I made mention of this paper regarding “pragmatics”: https://arxiv.org/abs/2210.14986 , again pointed to by the same Astral Codex Ten blogpost (where is that blogpost ???) [“First came Syntax, then Semantics. It looks like Pragmatics will be the next to fall.”] Prima facie it doesn’t seem particularly clear how the insights within said original paper from October 2022 could be translated into concrete engineering outcomes, but having had a look around, I found this very interesting paper, that reminded me of the renaissance in interest in symbolic AI, as well as providing context as to why solving pragmatics is significant (emphasis mine):
A hallmark of intelligence is the ability to use a familiar domain to make inferences about a less familiar domain, known as analogical reasoning. In this article, we delve into the performance of Large Language Models (LLMs) in dealing with progressively complex analogies expressed in unstructured text. We discuss analogies at four distinct levels of complexity: lexical analogies, syntactic analogies, semantic analogies, and pragmatic analogies. As the analogies become more complex, they require increasingly extensive, diverse knowledge beyond the textual content, unlikely to be found in the lexical co-occurrence statistics that power LLMs. To address this, we discuss the necessity of employing Neuro-symbolic AI techniques that combine statistical and symbolic AI, informing the representation of unstructured text to highlight and augment relevant content, provide abstraction and guide the mapping process. Our knowledge-informed approach maintains the efficiency of LLMs while preserving the ability to explain analogies for pedagogical applications.
Here is a Youtube video that touches on Neuro-Symbolic AI in the context of leveraging vector databases (such as Marqo) for RAG (retrieval augmented generation):
From AI Crowd: the commonsense persona grounded dialogue challenge, based loosely on this paper advertising a new dataset / set of experiments called PeaCoK (source code, code from earlier project). “Sustaining coherent and engaging narratives requires dialogue or storytelling agents to understand[ ]how the personas of speakers or listeners ground the narrative. Specifically, these agents must infer personas of their listeners to produce statements that cater to their interests. They must also learn to maintain consistent speaker personas for themselves throughout the narrative, so that their counterparts feel involved in a realistic conversation or story. However, personas are diverse and complex: they entail large quantities of rich interconnected world knowledge that is challenging to robustly represent in general narrative systems (e.g., a singer is good at singing, and may have attended conservatoire). In this work, we construct a new large-scale persona commonsense knowledge graph, PeaCoK, containing ~100K human-validated persona facts. Our knowledge graph schematizes five dimensions of persona knowledge identified in previous studies of human interactive behaviours, and distil[l]s facts in this schema from both existing commonsense knowledge graphs and large-scale pretrained language models. Our analysis indicates that PeaCoK contains rich and precise world persona inferences that help downstream systems generate more consistent and engaging narratives.”
From the world of WASM:
“Better things aren’t possible” feels like such a failure of imagination.
I’m reading the wasmtime 2023 report, and among the myriad of features they added one of the things they talk about is how they used formal methods to systematically eliminate miscompilations.
Like let this sink in. This is a (critical!) bug in a from-scratch LLVM alternative, built using bleeding-edge designs, for an in-progress language (Wasm/WASI), built by a handful of people. And they did it
Blender 4.0 is out, as is Godot 4.2. Both are notable project milestones. Godot 4.2 finally closes the gap with 3.5 by bringing in ANGLE, which allows compilation of artifacts for low-end systems.
Apparently GPT is becoming a Turing machine, according to this paper that proposes a new pattern it coins as “IRSA”: Iterations by Regimenting Self-Attention. The idea is essentially to structure prompts in three parts: first, by giving the LLM at least one highly explicit example. Second, prompting with fragments of execution paths from the aforementioned example(s). And third, explicitly forbidding (skipping) self attention to various parts of the text. Here is an example of this approach given in the paper for bubble sort ; evidently this is the sort of thing that could potentially be made programmatic on the prompter’s end:
As of the close of 2023, it seems it is now possible for state of the art systems to generate several seconds of quite good quality footage – to a prompt: Stability.AI: Introducing Stable Video Diffusion. See also this closely related work from Meta: Introducing Emu Video and Emu Edit, our latest generative AI research milestones. One of the more vocal advocates for focus on alignment has indicated where he thinks this might lead within 2 to 4 years (twitter). I suspect we’ll be there (minute long sequences of prompt-generated photo-realistic video) in no more than one and a half years, and possibly within one.
Today I figured out how to solve the perplexing “cannot send to unconnected peer id 1” issue, the basic problem was that I did have the peer connected to the multiplayer_api instance I had associated with my Networking global, but in Godot 4, networking is handled at node level.
Therefore, to fix, I just needed to point the multiplayer.multiplayer_peer of the node to the webrtc_multiplayer_peer on the Networking global. Now, no more red console messages!
However, I’m still not seeing things move in response to action by the host within a peer. I think this is because of at least one of the following reasons:
I need to set the multiplayer.multiplayer_peer for the remote nodes as well,
I haven’t set the multiplayer authority of my remote node properly,
I’m not adhering to magic new naming conventions in Godot 4.0.4
I suspect that the answer is not the third point, since I already added boilerplate before to fix a related issue with the new magic naming conventions.
Hopefully tomorrow I can narrow things down a bit further and close the loop on this.
Today I determined that on join of a peer to an existing instance with a host, in the joining client, func _on_connected(id) in my “multiplayer_client.gd” script is not activating for the host, just for the peer. So the peer gets the id for self, but not the id for host …
It is possible potentially that this is a comms bug, but it bears further investigation first. Is _on_connected something that should be run for the same id as the joining peer? i.e., should the peer with id 3 run _on_connected(3) ? Or just _on_connected(1), where 1 of course is the id of the host?
These are good questions, I will continue looking into them tomorrow.
Added a global Network object for the multiplayer_api,
Fixed my tooling for redisjson for arm64 architectures by using redis/redis-stack,
Used the new Network object for networking
I’m still encountering the error:
E 0:00:05:0053 CameraOrbitRig.gd:91 @ _process(): Attempt to call RPC with unknown peer ID: 1. Method/function failed. modules/multiplayer/scene_rpc_interface.cpp:311 @ _send_rpc() CameraOrbitRig.gd:91 @ _process()
… and many similar ones of same-said ilk on rpc_id calls though, I’m not entirely sure how to fix. The issue seems to be when I try to make an rpc call to the server of a webrtc mesh in Godot 4.0.x, maybe this is a bug and fixed in later versions?
It might be worthwhile bumping to Godot 4.1.3, I’m not entirely sure that the 4.0.x branch is sufficient for my purposes. But I’ll persevere a bit with this I think, I’d like to identify more clearly where the problem lies first. Most likely it is user error on my end.
Where's my hat?! 