So recently ZZZero did an awesome stream versus bots, using the SCHNAIL client. Give him a follow! think it went very well, showcasing the client, and the abilities of the AIs as well. This is just a short opinion piece.
The games
First, because that’s the first question on everyone’s mind: He only lost one game, when playing protoss (not his main race) vs. adias, which is the bot made by Samsung. However, I feel there were close matches, and great engagements. I don’t want to analyse all the games here – that’s more of Jay’s area anyway – but I do have some takeaways.
Don’t take this as defending the bots, or disputing the results in any way. ZZZero won fair and square every time. There was only one instance when a bot was bugged out, but that’s the bot’s fault, and it was losing anyway.
A common theme was ZZZero just plain outmacroing his opponents. This was very visible against Locutus, where the goons microed well, and traded favorably – but in the end, ZZZero just had more stuff. Many bots were turtling a bit, taking only one or two bases, and that was their undoing in the end. Many of them had weaknesses to a specific tactic (like Hao Pan was reacting sluggishly to muta harass with his Goliaths, or other bots not really handling reaver drops well), but the games were entertaining, and the victories weren’t too easy. Few bots are built with vs. human play in mind – perhaps that will shift a bit after this. An absolutely hilarious, but ultimately ineffective tactic was trying to play the Stove by PurpleWave.
I think this just shows that the bots have their own meta, and react to each other. I guess some of the learning was made with that in mind as well. And let’s not forget that ZZZero is a pro – I think less skilled players might have some surprises.
Finally, I must commend the fact that none of the bots crashed, or slowed the game down. Granted, I filtered them beforehand, and we only saw top bots being played, but still. Krasi0 bugged out in one match, but that’s a different issue. I think the authors did a good job.
Another recurring thing was that new viewers were routinely surprised that these were AI. I guess they expected to see vs. stock AI matches. I’d like it to be more clear in the future, but maybe this is just a public awareness thing that yes, these AIs exist, and yes, they are becoming quite good. The bot scene definitely exists in a bubble now, and some things that are maybe evident to authors (or just plain old developers) need some explaining.
About the client itself
Sidenote: ZZZero was using an older client (0.1) – I gave him a new release that day, but he had trouble setting it up, so we agreed to use the stuff that worked for sure. Such is the cycle of software development.
So for the most part, it worked well. The only thing that was annoying that the StarCraft instance had to be closed manually, otherwise it would jump to hosting a new game. Since then, I fixed stuffz, and added some visual enchanchements. This is a current state screenshot:
I was analysing the cursor movements a bit, maybe the play and the Quit/Log out buttons should switch places. It was my intention that you only need to do stuff on the left side of the screen if you want a game.
About bot ratings
One thing that often comes up is to able to sort the bots according to their score. Now, there are a couple issues with this. The SSCAIT score is a good indicator, but by no means reflect the order of difficulty for humans – that is calculated by . Also, technically, there is no way I can retrieve that from the SSCAIT page right now. There is only a score value which is calculated by the number of wins or something like that. But in any case, as things develop, I don’t want to rely on SSCAIT data and infrastructure – it’s not fair to them.
So that brings us back to the human vs. AI score problem. Some of you might say “jUsT uSe ElO”. Alas, it’s not that simple. The ELO system assumes that every participant can play every other participant. In SCHNAIL, it’s not the case, the bots are not meant to play each other there. On the SSCAIT and BASIL ladders, sure. So just combine the results, and calculate, right?
Well, the bot ladders are 0/24, and on BASIL, the games are played in headless mode, as fast as they can. That means I have an order of magnitude more bot vs. bot games in any case. I could just take a sample of those, and throw it in the mix for vs. human games, but I really need to devise some system for that, and consider both game types in equal amount. And this might be just plain not working when people upload bots, or versions of bots that only (meant to) work in SCHNAIL. (Not to mention that I depend on the availability of these servers, which I don’t much like).
BUT WAIT? Couldn’t I just give bots an ELO to start with, then have humans play against them, thus giving a realistic estimate? Well there are multiple problems:
- Bots don’t really gg out and quit, so you need to kill the last building in the corner to actually win. We saw in the stream that ZZZero just said “ok, this is gg” and quit. Technically, that counted as a defeat, and bots have no way of telling that.
- So I just need to add the ability to gg out. Except what will that count as? And how can I tell if a player is using it to avoid a defeat? Even if i count it as a draw, it is not the best.
- Okay, so let’s have a casual and ranked mode, and in ranked, if you gg out, that counts as a defeat. Except that can be gamed pretty easily. Alice picks a weak bot, ggs out 50 times, then Bob just defeats that boosted-ELO bot a few times, and Bob is now on top of ladder. Come think of it, you can do this even without gg-ing out, but it would take longer. To avoid this, maybe ranked mode can only be played vs. a random bot? That would make this tactic infeasible.
It seems I can’t really remove the ability to game the system. However, a thing I can do is remove the incentive to do so. What if I would rank the players, but simply don’t display that score anywhere? So you could game the system, yeah, but you don’t really get anything out of it. Maybe this – and a random-ranked ladder – will be the way forward.
This bothers me, but it’s not the most urgent thing to do about SCHNAIL right now. Bringing up the infrastructure, and having it available to a wider audience is. Believe me, I’m working on this as much as I can. Please consider helping me making it happen by supporting the project on Patreon! And of course, follow me on any social media you like for updates. Thanks for reading!