State of the Game #272: Digging Deeper than a Europa Miner

Hey everyone, sorry for the later State of the Game post, I have been in my own personal jail of debugging! 🙁

But really, it’s a good thing! The great thing about digging deeply into a set of code is that you can fix things you might have overlooked before, as well as all the monitoring that I have added has allowed me to identify even more fixes!

To back up a little bit, the rabbit hole of a bug that I am talking about is the “9FPS” bug. This bug is officially a heisenbug only showing up at random times and most frustratingly, NOT showing up when I have a performance profiler attached. This means that to debug it, I make a change, then spend about an hour doing testing to see what the change did when the bug occurs, then make notes and repeat the process. This has lead me through quite a bit of the code base, as I test red herring data, isolate it, then figure out it’s a symptom of the issue but not the root cause.

Good news!

I have, during this process, found and fixed two other issues that were fairly serious bugs. One issue had to deal with circular logic when doing dynamic re-attachments on component destruction. Since each part in M.A.V. needs to be attached to something, when a single part explodes the parts that are attached to it try and find a new overlapping attachpoint and attach to that new parent part. In example, a cluster of weapons usually has each weapon overlapping at least 2 different attach points. This bug would happen when this dynamic attachment would create a circular graph, instead of a nice tree structure like it should. This would result in a recursive, infinite loop, that would eventually lead to a stack overflow and hard crash to the desktop [normally proceed by a massive lag spike]. This issue has now been fixed by doing a proper cycle check when rebuilding the tree and not attaching to nodes that are in the process of attaching as well!

Another big performance hit came from having 5+ turrets deployed at once. This one started to rise in occurrences with turrets being used more, which also means the AI uses them more, which made this issue a spiraling performance hit. When this bug hit, frame rates would start to progressively drop and eventually this would result in a similar situation as the ‘9FPS’ bug, only getting better when the turrets were destroyed. This was actually caused by a combination of things, but was only triggered once the Heat System was added to the game. In MAV code, each part has a ‘design’ class and a ‘data’ class. The design class can be many things, from the weapons to the cockpits and contains all the good stuff that you care about. The data class is universal and contains all the good stuff that the game cares about, like how to save and load the part, how to attach them, how to apply camo, and that stuff. Because of the way turrets were built, the turret class was not getting a proper data class. This, actually didn’t matter for ~4 years. Then the heat system came in, which uses the data class to track the last person that did damage, so the destruction of the part, even if done through heat, can be credited to the correct person. Now, this means there was a call every update that was trying to access a data class that wasn’t getting created and throwing an error! The error was being processed by the custom logging system in MAV and while it wasn’t dumping into the log file, it was clogging the logger system with hundreds of logs per second, which would case the frame rate issues. This only got worse as the turret started taking damage, as each damage event would trigger more logs as well!
So, some progress, but more to dig and more to go! Performance is becoming a bigger and bigger part of my focus and will likely seem some tweaks in the April update. I have many other items planned for April, but what good will it do to spoil the surprise now? 🙂

See you next week!