Multi-streaming setup for BSL, and a big box of tentacles

So it all started when my good friend, ZZZero, the organizer of the Bombastic Star League asked me if I can help him out with something akin to squad streaming.

(If you are not that familiar with the scene – it’s a StarCraft: Brood War championship league, with high production value – with good commentators, and lots of hours of entertainment. Go watch them!)

The basic problem BSL faced was during the shows, that they wanted a way for casters to easily switch between player POVs. Ordinarily, the players could just stream to Twitch, but that has some delay in it, and that could affect the narration of the games. Also, while it is certainly possible to import those in OBS, or any kind of broadcasting software, it is not the best solution. When using OBS, the caster kind of has to show what he is watching – and we talked about a way to avoid that. Another use case is a foreign language caster (Foreign in this case meaning non-English, as opposed to non-Korean) might want to import the player POVs without commentary, and narrate the games in another language. Talking over an existing cast is not really preferable.

This is the background so far. ZZZero is not a technical person, so he asked me to do some research, and I gladly obliged. I started looking at how to set up a custom RTMP server.

Turns out it’s not that hard, I found a guide which I immediately tried out. This is basically just an nginx instance running under some Linux environment. I set it up using one of the dev servers in the cloud, we tried it out, and it worked without a hitch.

The only real bottleneck in this case is the bandwith. Nginx in this configuration is really light on system resources – I briefly looked at memory, CPU, etc. usage, and indeed, it was negligible. (I used one of the SCHNAIL dev servers to test, and it didn’t really cause any problems). However, as an engineer, I always have to plan for the worst.

Generally, upload is much less for home internet packages than download speed, so that would be a real bottleneck in any kind of residental internet package. Streaming in 1080p consumes 5 Mbps, while 4K uses up 25 Mbps. Let’s say we have 2 players, 2 casters, and one producer as a minimum (we saw this setup in BSL previously, so it’s not unrealistic.). Suppose we want to see the casters themselves every now and then – so they have to provide a video output as well. Just this simple setup involves the following:

Assuming everyone streams in 1080p – and nowadays, viewers are kind of expecting that – that’s 4×5 Mpbs upload, and 10×5 Mpbs down (20/50 Mpbs). While this is not that much in itself, you are probably using your home internet for other purposes, or sharing with others. Also, outages and lagspikes happen all the time. While It might be ok for you to not look at cat videos for 5 mins [citation needed], it’s not acceptable for a broadcast.

Renting a dedicated line for this would be prohibitively expensive, so I started looking at cloud-based solutions. Turns out, they are the ideal use case for this, as bandwith/data usage are handled by default. I chose an Amazon Lightsail instance with Ubuntu running on it, and set it up. Yet again, this wasn’t a particularly hard part, it worked well.

So far, I have been following the guide I linked before – but that solution does not have authentication. Meaning anyone who knows the IP of the server can just stream to it anytime. This is obviously not ideal, as data usage costs money. Also, someone can hijack the stream – and even though this is not something that is open to the public, once the IP address is revealed, the risk is there (and a disgruntled former player/caster/etc. is not unheard of). Security by obscurity has never worked on the internet before, so I won’t be counting on it now either.

I found a relatively simple solution here. You can just add a hook into nginx that verifies the URL using a POST request. The example there uses a php page, but there can be anything that returns the proper status codes. First, I just wanted to use something quick and dirty that returns an OK if the stream key is in a predetermined list (like a text file).

But the disgruntled player problem is still present – if someone can leak the IP, he can leak the stream key as well. I can just create new ones of course, but that requires active administration from my part, and I don’t want to do that. The next logical step is to give a way to the tournament organizer (Mr. ZZZero, in this case) to handle these. Okay, a simple SPA could do the trick here.

But then again, at this point, why should we have one user (or two)? I decided to have a fully-fledged application that can handle this for multiple users. If I’m gonna rent a cloud server, I should damn well utilize it!

Enter the Tentaclebox.

At the time of writing this, it is under active development – I planned it as a fairly simple SPA (Single Page Application), where you can manage access and see how many in/out streams are present at any given time. I just used a Spring Boot application for this, with an Angular frontend – probably something more utilitarian-looking. Presentation does matter however, so a plain text page is just not enough. But that can wait.

I’d like to thank MasterRey here for helping out with development as well.

We had some testing, then BSL10 used the setup in production, and it worked quite well. First-person views shown on the streams were well-received, and the foreign language casters could pick the feed they wanted to commentate. There are some observations though, and I need to look deeper into this to solve some of this. Theser are the following:

  • The endpoints are still not secured well enough, and if you know a certain player’s feed address, you could technically tap into it. This is a very low risk as well, you do have to know about it, and we just assigned random addresses. It is mitigated by good opsec, but we still have to plan for the worst. Also, one player can just look at the other’s feed, which is not ideal.
  • There is a lag when loading and switching between feeds, and currently they are a few seconds out of sync from the main stream. This is not a big problem, but I’d like them to be more in sync – potentially you should be able to look at the game, and both PoVs at the same time.

These all are improvement points, and not bugs – the core functionality performed flawlessly, and I’m really happy it did.

Thanks for reading! I hope I can help some tournaments, or any other multi-streaming setup with this.

If you liked this article, feel free to follow me on any of the social media channels (links are on the top bar too, but here they go: Facebook, Twitter, YouTube, Twitch), or maybe even consider giving me a dollarydoo on Patreon!

Leave a Reply