Welcome to another State of the Game! The weeks seem to really be flying past these days.
This past week I have been tracking down a ‘Mystery Bug’ and I would like to tell you a bit about the hunt. Normally, I don’t like to get very technical in the State of the Game posts, but since this has been a bit of an ongoing hunt, I wanted to detail it a bit, since it’s possible you got affected by the bug.
It all started with the release of version 0.5.14. Suddenly, when players updated to the latest version, all they got was the M.A.V. logo, then a black screen. Nobody could get past the black screen, and a worst case situation had risen. Nobody could play the game.
I went into immediate damage control, trying to reproduce the issue, but everything was working fine in the game editor. I tried in the cooked build, and again, everything was fine. No black screen, everything worked. Then I tried again, but with the game installed. Bingo! Black screen and an unplayable game. So I zipped up the uninstalled version of the game, added it to the User page with a message about the broken installer, and went into debugging mode.
The First Head
When you first start up M.A.V. a few things are happening in the background. The first thing that happens, is the game spins up the updater program to check for newer versions of the game. This is a separate process than the game, because it has to shut the game down to do the update. Because of this, there is a communication system in place for the update process to notify the game when it is done checking for updates. This notification was the first head of the hydra bug. Suddenly, this notification was not happening, and the game fell into an unhandled state, infinitely waiting for the update. Yes! I had found the issue, so I made the changes to wait for the notification, or after a certain amount of time, cancel the update and move on. I updated the installer and everything was good. But why would that code suddenly stop working?
A New Head Arises
Snapshot versions 0.5.14_1 and 0.5.14_2 both went out the following week, adding a few new features and polish to the game. Then, strange bug reports started to come in. Mainly, in multiplayer games, things would just stop working entirely. Servers would mysteriously disappear, and the strangest of them all, players would get locked into place, but only on the server and other computers.
You can actually see a lot of these bugs happen during this video.
[Also, videos like this REALLY help me find issues. The videos by LegacyElite84 have helped me track down more things than I can list!]Well, this was quite a mystery now, because this is the second case of ‘untouched’ code starting to fail in new ways. I checked the automated bug log to try and see if it caught any code errors during the recording of the video and it was completely clean. I went back to the beginning and started tracking on the only clue I had, a single function that failed to get called 100% of the time during the check for updates section.
A Breadcrumb!
I setup my debug environment with as many reporting and logging tools as I could, created the debug builds, and started triggering the update. Nothing. Not one of my tools could catch a single thing that would be preventing the function from being called. It was like it didn’t exist. As I was trawling through the Unity Debug log for about the 15th time, something finally caught my eye. 3 simple lines, in a place they shouldn’t be:
Fallback handler could not load library C:/Users/Chad/Desktop/Build/Mav_Data/Mono/libc
Fallback handler could not load library C:/Users/Chad/Desktop/Build/Mav_Data/Mono/.\libc
Fallback handler could not load library C:/Users/Chad/Desktop/Build/Mav_Data/Mono/libc
Well, that is not supposed to be there… But what would cause an entire library to not load? I contacted Unity support at this point, to see if they had any insight into what could be the issue. Sadly, this was the first time they had ever seen this happen. Ever. So I was alone, in uncharted waters. Time to start digging.
The Hunt
M.A.V. is currently ~9GB in size, with all the artwork, textures, and assets. It has about 70k lines of code. Somewhere in that, something was causing issues. I branched the full game over to a debug folder and began chopping away. I removed all the game play levels, the garage, all the game play code, all the code for your MAV, slowly but surely removing more and more stuff. Finally, after several hours, I had isolated the bug to a single area. Ironically enough, it was within the automated bug reporter.
Suddenly, things started to make sense. I use a great tool by GameAnalytics.com to do automated reporting of critical errors. It also provides some great design feedback, seeing how people are killed, what weapons are doing the most damage, ect.. They updated their API in December, and I updated to their latest version of their tool, right before the 0.5.14 release. None of the issues were caught in the snapshot build because they didn’t get triggered until the update happened.
The Fix
With the problem tracked down, it came down to finding a solution. I opened up new bug reports with both Unity and Game Analytics and both companies were able to confirm my findings. I tried hunting through the code, selectively removing calls and references, trying to get a better track on the true failure trigger. Unfortunately, time was running out. Players were getting worse and worse bugs, some of them even making the game completely unplayable. During testing, I hit a bug that made the game render a 1 frame per 5 seconds [yes .2 FPS!] if there were flame thrower particles on screen. I had to make the hard choice and pull the Game Analytics code completely out of M.A.V., at least until a solution was found.
This bug hunt is not over, but at least the damage is contained now. I hope to hear back from either Unity or Game Analytics with either a solution or a lead. Until then, I will continue on the things I do have control of. If a solution doesn’t come in the next few weeks, I will have a custom system that I will create and put in place. This is likely to happen in the future anyway, as I think the data coming is very interesting and I would like to be able to make it available to players, in a web enabled format, similar to how Halo had the game viewer and stats available online.
That concludes this week’s State of the Game! I hope to see you back next week!