Today I groomed the backlog a bit for the sandbox project, closed the last sprint, and started a new one.
The goals for the February sprint for the sandbox project were:
I make partial progress on two of these.
The goals for the March sprint for the sandbox project will be:
Generally optimistic. As I write this on the 24th of September 2022 things have been progressing well.
2 All the ways of a man are pure in his own eyes,
Proverbs 16:2-3, ESV UK
but the Lord weighs the spirit.
3 Commit your work to the Lord,
and your plans will be established.
This month I:
Slightly slow this month, largely due to wrestling with third party codebases and seeking to understand them.
Things were a bit slow this month, but I did do a fair amount of careful thinking about how to deal with memory issues in comms, as well as some basic security concerns.
I’ve realised that likely I face a significant setback, in that in order for comms to do what I want, without me needing to
1) plug a large number of antipatterns + memory leaks,
2) convert it to docker & deploy via kubernetes and
3) likely spend a small fortune on k8s replicas in order to service a relatively tiny amount of requests,
I should strongly consider rewriting it in Golang. I may still try to do the above, but I think it is rational at this point to start investigating a rewrite and implementing some basic functionality in one. So I will start looking into doing these things in parallel.
Another matter which had a mixed outlook was the sunken cost on coturn. I did however make some inroads that should inform some issues I have with compiling protongraph, but at least now I know that I should switch to a different and more modern turn server.
Reasonable.
My basic plan is:
Per the roadmap, to slightly extemporise upon the points from here:
Near term (from now through to the end of the April 2023 sprint).
Not so near term (from the May 2023 sprint onwards through to December 2023, i.e. likely Q1 ’23 -> Q2 ’23 in real time).
Deferred
The setback I’m facing should hopefully not slow down things too much, and I should still be able to get to alpha by June 2023 in actual time.
Narrowing things down along the lines of this post, the main pieces of complexity remaining to solve for are these:
One of these (payments) should be more of less done by the end of the next sprint.
150 points should be actionable in theory. Assuming that I end up needing to do another 200 points, and I get through about 20 points per sprint on the lower side of things, that means I will need 10 more sprints to get things to “alpha ready” state. That basically will take me through to the end of the December 2023 sprint.
Taking into account an additional margin of 2 months on top of that, that should see me getting to alpha conservatively by the end of the February 2024 sprint. i.e. another 12 sprints. If I can increase my lead by another 3 months (from the existing lead of 5 months to 8) by June 2023 in actual time, which is potentially doable, I should be alpha prototype ready by the end of June 2023 in actual time.
Anyway we’ll see how it all pans out in the wash.
I’ve continued looking at coturn and, although I managed to get unauthenticated stun requests working locally again, I continue to be perplexed by how to do security with it. Looking on the web I found this alternative written in golang: https://github.com/pion/turn . A bit more modern and, although fairly low touch at present and less battle-tested, potentially has the features I am after – namely, security for a turn server.
With regards to coturn, –secure-stun which is a flag that is apparently supposed to secure coturn doesn’t work, and apparently is “blocked” on this mozilla bug. Not sure that I buy that though.
The coturn code is very complex, very old, and very messy, i.e. legacy code. Which makes sense as it is an old project. It is old on github, and existed for many years before that – additionally, as I understand it, the current maintainers are not the original developers who wrote the project. I’ve worked on legacy systems myself before. I know how difficult they can be to refactor and change, to tidy up and to modernise. Legacy systems are often battle-tested and widely used, and often the existing userbase just wants basic maintenance patches, they don’t want any major improvements – or any form of significant change at all really. So I empathise with the pain faced by the current maintainers.
Certainly though, services written in legacy code are not necessarily a good choice for a greenfield system, such as what I am building.
Just because something is legacy, however, should not necessarily be a show-stopper to using it in a new project. And I did have a good go at it. But disabling the stuns and stun lines in “get_default_protocol_port” within ns_turn_utils for coturn doesn’t block unauthenticated stun as I wished – and there seems no obvious documented or supported path to making authenticated requests, or even an established pattern to secure coturn – at all – despite multiple questions over many years by developers in the coturn issue github knowledge base, which have been more-or-less unilaterally dismissed by the maintainers. Very confusing and a tad perplexing.
I’m put in mind of one of the seasoned developers I know who told me once: “If a tool doesn’t do what you need, it is not incumbent on you to fix it.” I guess my slightly more admissive take on this is that: if a tool doesn’t do what I need, and it is not moderately straightforward to fork and fix, then maybe it is worthwhile looking into using a different tool.
The pion/turn docs on secured requests do look very straightforward. I might try giving this project a spin and see how I “go”.
Today I:
However the WebRTC connection from the client is not working for stun, turn, 5349 or 3478 ports. Puzzling. I suspect some sort of caching issue, because the test node process can connect easily enough via stun on port 3478. Confusingly, the remote coturn process doesn’t seem to work now for turn connections, so definitely some form of caching issue I think. Whether that is in the WebRTC module within the client or elsewhere, that will require investigation + further research.
Maybe I can test turn connections with username and a credential by leveraging telnet?
So still to resolve:
With regards to the malformed archive problem, it appears to be due to the fact that I have one or two packages that are “thin” libraries, and some are in ELF 32 vs ELF 64 format when being linked together with the ar tool, see here. One workaround is to run the ar tool on each .a file individually, see here and also here.
For the record:
https://bugs.chromium.org/p/webrtc/issues/detail?id=5022#c8
for lib in `find . -name ‘*.a’`;
do ar -t $lib | xargs ar rvs $lib.new && mv -v $lib.new $lib;
done
These command can convert thin libraries to normal libraries.(https://stackoverflow.com/questions/25554621/turn-thin-archive-into-normal-one)
This probably doesn’t need to be fixed, maybe I can give this a miss – as something similar cropped up when building protongraph.
Today I:
So this is probably good enough, I’d still like to sort out logs though. And I am a bit nerd-sniped by the fact that turn-only is “bad configuration”, why is that? After renaming this to use-turn-only, still the same problem. A bit stumped!
Setting “verbose” in turnserver.conf gives logs, but now stun and turn don’t seem to be working in my client … confusing. Probably a cache thing, I’ll look into this again tomorrow.
So, still to-do:
I’ve been reflecting on the comms memory leak issue, and, although it is possible that I could address some of the anti-patterns that have led to this matter, I think the issue would remain – comms needs to be a highly performant service, and it needs to be robust from a garbage collection and usage of memory point of view.
For this reason, I think it might be necessary to rewrite it in a language that lends itself to better performance and robust garbage collection. Choices for this could be C++, Rust, or Golang.
I haven’t programmed in Rust, and C++, although very modern, has its downsides, particularly with memory leaks of its own. Although one can resolve the issues along such lines in C++ with libraries such as Oilpan, I think the better choice is Golang. Besides, many of the systems I’m deploying (a couple of the consumers, coturn, the client) are already written in C++, so some variety would be useful to play with from a modernisation point of view.
However, I probably won’t go “all-in” on a rewrite just yet. But I think starting to explore what such a service could look like, to the point where it can facilitate Coturn OAC handshakes is probably a good idea. I’ll aim to look into this over the next couple of sprints.
I guess the learning here is that node.js is a good option for quickly prototyping a system, but perhaps slightly suboptimal when aiming to build a robust and high throughput production service.
So I found the docker images shipped with coturn not entirely helpful as I couldn’t build them straight out of the box. It seems that docker-compose is the name of the game with coturn, but I don’t want to run 5 or 6 containers, I just want to run one.
I had another look and started playing around with a small node service in order to check the connection, things work with this code, leveraging this package: https://www.npmjs.com/package/stun.
const stun = require('stun');
// stun.request('stun.l.google.com:19302', (err, res) => {
// if (err) {
// console.error(err);
// } else {
// const { address } = res.getXorAddress();
// console.log('your ip', address);
// }
// });
stun.request('0.0.0.0:3478', (err, res) => {
if (err) {
console.error(err);
} else {
const { address } = res.getXorAddress();
console.log('your ip', address);
}
});
Sure enough with stun:0.0.0.0:3478 the client could connect too, so evidently coturn is doing what it should. Port 5349 doesn’t work though, i.e. doesn’t facilitate networking properly, I think that’s because I have an issue with certificates. Of course I should doublecheck turn:0.0.0.0:3478 username:password before proceeding.
I did need to expose the ports via docker run -p 3478:3478 -p 3478:3478/udp -p 5349:5349 -p 5349:5349/udp -p 49152-50000:49152-50000/udp coturn , I might want to script that properly. Also scripting jumping into the running container and/or stopping it, I should do that too.
I’m not getting logs generated though on connection, and turn-only is still bad configuration despite my best efforts.
And I will want to front the whole thing with nginx eventually when I deploy it via terraform, as it will be running in docker on the target VM.
So: certificates, logs, new configuration not working, helper scripts. And the question of modifying the existing terraform deploy script so as to leverage nginx and docker, but that can wait for now.
Regarding logs, probably what is happening is that the user running the coturn process doesn’t have permissions to write to the log file. Now I am creating a user called ‘turnserver’ and giving it those permissions, but I’m not sure I’m running the executable as it. Should be easily fixed by inserting -u turnserver in the appropriate spot:
CMD ./usr/coturn/bin/turnserver -c /etc/turnserver.conf -u turnserver --pidfile /run/turnserver/turnserver.pid
Certificates are not critical, as the whole thing will be fronted by nginx eventually.
The new configuration matter is a bit more serious, it would be helpful to unpack why things are not working properly there.
Today I discovered that, well, there is already code written to do more or less what I want: https://github.com/coturn/coturn/tree/7486e503748b8d8ca45b4e555db7ff62ec850ffd/docker/coturn .
All I need to do is to hack on top of that.
Tomorrow I’ll pick up the pieces of what I’m working on and switch to leveraging either the alpine or debian docker builds.
Today I continued plugging away at trying to get coturn to work locally. I discovered in particular that it seems that any port works for stun or turn locally, so evidently my initial success was perhaps not nearly quite!
I managed to fix a couple of problems such as connecting to sqlite within the docker process. I also spent a large amount of time trying to fix an “openssl/md5.h” not found error. I discovered in testing that ‘turn-only’ is bad configuration.
Bad configuration format: turn-only
Overall results are as yet inconclusive.
Update: having continued to wrestle with compilation a bit more, I’ve started to conclude that maybe I should look into introducing scons tooling to compile, following my experiences with protongraph. In particular I shouldn’t have to mount paths to libraries in osx if I’m compiling for linux; such results in nonsenses such as needing to get systemd/sd-daemon.h from osx, but of course this is a linux only thing.
Where's my hat?! 