Working Around A Bug In Unity’s Input System By Building For Release

Edit: Someone from Unity Tech. found this post and is working on a fix.  So that’s good 😀

This post could also be titled “Why I Absolutely Hate Closed Source Game Engines (And I’m Totally Not Talking About One In Particular)”.

This weekend, a couple of friends and I participated in the Game Maker’s Toolkit 2018 Game Jam*.  As part of it, I dusted off an input system I wrote for a local-multiplayer Unity game a few years ago.  Since it uses XInput for Xbox controllers, I thought – “Hey, I have an Xbox One set up for devmode and I read somewhere that UWP (Universal Windows Platform) games now have unlimited resources…  Wouldn’t it be neat to port our game jam game to the One?”  Yeah, that would be neat, and being able to eventually tack “Shipped Xbox One Indie Title” onto my resume would be pretty cool.  Unfortunately, it wasn’t quite that simple.  So I’m writing a blog post about it, mostly because I want to rant about the last 18-or so hours of my life.  Bleh.

* We missed the deadline by ~5 seconds due to an issue in our itch.io page.  Turns out you need to actually publish a game if you want to submit it to a game jam.  Bummer, because our game was pretty neat 🙁

No XInput?

Given that Xinput is tied to Xbox controllers, I assumed that it was still supported on Xbox One.  I had never done any development for UWP, but to my optimistic mind, that was a pretty safe assumption.  So the first thing I did was simply change my build settings to target UWP (which, to my surprise, now used IL2CPP by default – this will be important later).

After a lengthy build process in Unity and an equally lengthy build in Visual Studio, I ran it on my desktop and, lo and behold, it… crashed…  It looked like it was crashing in XInputDotNet’s pinvoke code, so I tried replacing the version in our Unity project with the latest release and… it all worked!

“Since that was so simple, it shouldn’t take much more work to run it on Xbox,” I naively thought.  So I configured Visual Studio for remote deployment (to any Visual Studio devs out there, maybe I’m in the minority, but this process is ridiculously unintuitive to me) and…  We have a loading screen…  and a main menu!  With… no input…  Ugh…

After a bit of searching around, I found that Microsoft has more or less abandoned XInput in favor of the Windows.Gaming.Input namespace.  Just to verify, I made a new Unity project and made a script to move around a sprite with the new API, deployed to Xbox, and it worked

It is a pretty well engineered API and, from what I can tell, should support gamepads other than just Xbox.  Unfortunately, it isn’t supported by Unity standalone builds, so we can’t just use it everywhere.  Drat.

I started combing through the internals of my several-years-old input system and realized that it was a horrible mess.  It would have taken at least as much time to work the new API into the old system as it would to just rewrite it, so I opted for the latter and implemented code to conditionally use Windows.Gaming.Input rather than XInput if UNITY_WINRT was defined.

At some point during all of this, I switched to the .NET scripting backend so that I could iterate on the hardware a bit faster as .NET builds are considerably faster than IL2CPP builds.  This will be very important later.

I deployed to the One, and after a bit of fiddling…  It worked!  It was horribly slow (compiled in Debug mode), but it worked!

And then my controller died, and the game crashed.

Huh…  That shouldn’t happen…  Maybe it was a one time thing…?  Time to reproduce it, and…

Nope, it crashes every time under those exact conditions.  But only if the controller is connected when the game launches.  If I disconnect the controller, start the game, and then connect/disconnect it, everything works fine.

Well this is going to be fun…

Fruitless Debugging

After fiddling with it for awhile, I went back to that minimal project I mentioned earlier and tried it with UWP on my desktop.  The same behavior occurred here, so at least it wasn’t an Xbox specific bug.  That sped things up quite a bit, as I could build locally much faster than building and deploying to the Xbox.

As I mentioned before, I was using the .NET backend for UWP.  Because of this, errors in native code seemed to be confusing the hell out of the debugger.  I found a combination of steps with which I could actually get the erroneous native callstack (which also required hunting down the Unity player symbols – why are these not loaded by default?), and it seemed to be breaking when attempting to access address 0x10.

Maybe that would be helpful to someone out there, but to me, that specific address just seems…  Weird…  It would be one thing if it was trying to dereference 0x0, but consistently 0x10?

At some point, I realized that if I never called any code from Windows.Gaming.Input, the crash didn’t occur.  At first I thought it might be a race condition involving querying the gamepad API, so I tried only calling GetCurrentReading() once at the beginning of the application’s lifetime, but that unfortunately didn’t change anything.  It was still crashing when attempting to access 0x10.

After a taking a food break, I decided to try moving back to my main project.  I was getting a really weird compilation error with the .NET backend, so I decided to try switching back to IL2CPP.  After another lengthy build process, I built the game in debug mode and the same error occurred.  The major difference, though, was that since I was debugging IL2CPP (native code), the debugger hit a proper breakpoint rather than the buggy mess that was .NET debugging.  Finally… progress!

It appeared that an assertion was failing in dynamic_array<GameControllers::Gamepads::XboxGamepad, 0>::erase(GameControllers::Gamepads::XboxGamepad*, GameControllers::Gamepads::XboxGamepad *),  from UnityPlayer.  This looked to me like an internal Unity call, but for all I knew, it could have been internal to the Windows.Gaming.Input library.  The assertion that actually failed was ‘input_end <= end()’, which looks to me like a bounds check that failed.

My best guess is that the Windows.Gaming.Input library will keep a gamepad around for a set number of polls after a controller has disconnected so that services can safely handle disconnects, but the fact that I’m calling the code as well as Unity confuses it, leading to a failing bounds check as Unity tries to read past the number of allocated controllers.  This wouldn’t explain why controllers plugged in after the game starts are bug-free, unless Unity allocates space differently for them.  However, since Unity is closed source, this is all guesswork.  I really have no idea, and short of going through the disassembled code or getting a job at Unity, there’s not much way for me to know.

“Oh well,” I thought, “I’ll submit a bug report and hope they get around to it.  Let’s at least see how the game performs in Release mode.”

So I built for release mode and, of course, tried to replicate the bug.  And I couldn’t.  It was gone.

Ship it.

The Case For Open Source Technology

My guess is that the bug disappeared in Release mode because it skipped over the bounds checking assertion.  Which, of course, means that the bug is still there, but it’s being silently ignored.  Assuming I was accurate in my assessment of the symptoms, this shouldn’t actually end up being a problem since it only ever occurred for the first gamepad.  I doubt this will cause any problems, but the whole thing still feels very bad.

No matter what, all of this highlights a problem I have with using Unity (or closed source libraries) in general – I shouldn’t have to blindly guess what the problem is and hope someone else fixes it.

Unity is a pretty decent game engine.  I have some gripes with it, but overall, it’s a pleasant enough experience if you do relatively simple things the way they want.  However, as soon as you start doing things that are slightly more advanced or otherwise do not fit in their framework, you start finding bugs and corner cases where everything starts to break down.  If the engine wasn’t such a black box, a lot of this bugginess might honestly be perfectly fine.  I’m very happy to drop down and try to fix the engine code if necessary, then submit a pull request.  You have happier customers with a more stable product.  Everybody wins.

Instead, I’m forced to fiddle around with snippets of barely readable, machine generated C++ code and guess what, in functions I cannot access, is causing a potentially production-stopping bug, only to have the bug disappear because the error-checking mechanisms are disabled.

What?!?!?!  How can this be acceptable of a professional product being used to ship thousands of games?

Something like this seems like it should be a fairly trivial bug in Unity’s gamepad logic or, at the very least, it seems like something that could be easily hacked around to prevent crashing on a pretty important platform.

But we can’t do either, and are forced to hope that Unity Technologies fixes the bug eventually.  However, it doesn’t take long searching through their bug tracker to realize that they don’t exactly have the best track record with maintaining existing features.

Whatever.  The project is working thanks to bugs killing other bugs, so that’s good enough for now.

/rant

Networking Woes: MTU and Packet Loss

Our game, Gravitas: The Arena, is using a custom high-level networking solution built on top of Lidgren.Network.  We’ve had a few lingering networking bugs for awhile now that have gotten buried under other priorities.  Today I sat down to try to solve all of them, and luckily enough managed to do so.

Aside from a few minor bugs, there were two remaining major issues that I wanted to address today:

  1. Intermittent, extreme packet loss over LAN
  2. Packets not coming through before a notable delay over the internet when using UDP hole punching (but not Hamachi)

Continue reading “Networking Woes: MTU and Packet Loss”

Adventures in C++ Hotloading: Changing Data Structures at Runtime

I’ve been working on a miniature game engine (lovingly called Mingine) as a little side-project lately.  It’s very barebones right now – pretty much just a platform layer based on SDL with a sprinkle of WinAPI, a mostly-functional graphics API abstraction layer (with an OpenGL backend), some image loading code, input handling…

I hate myself for this caption
The glClear is always greener somewhere else!

Oh, and it supports C++ hot-reloading, which is actually the whole reason I started Mingine in the first place.  In this article, I’ll be talking a little bit about the early state of that architecture, as well as some of my takeaways from it.  This will be a little longer than usual, so buckle up! Continue reading “Adventures in C++ Hotloading: Changing Data Structures at Runtime”

Extending Unity for Git (and maybe other source control?)

Note: The really interesting bit is at the end of the article.  So if you don’t stick around for the whole journey, at least skip to the end!  That being said, this is all useful stuff so you should, yeknow, stick around…

Unity is a pretty solid game engine that at least a few people are using, but it doesn’t play very nicely with Git – my preferred source control mechanism – by default.  While I can’t speak for everyone, there are three major issues I’ve always wanted to address:

  • The default setup all but guarantees that having two different users push the same scene, prefab, ScriptableObject, etc will render it totally unusable and force you to rollback (quite painfully) to an old version.
  • Git, by design, doesn’t play very nicely with binary data.  It’s diffing tool doesn’t support binary files and, as such, it stores the entirety of each revision of binary files for every commit.  As a lot of game content is stored as binary data – FBX files, textures, etc – this presents a bit of a problem for storage.
  • After playing with Unity Collaborate a bit, it became extremely evident that the UI/UX for interaction between traditional source control and the Unity editor could be drastically improved.

In the past we’ve sort of just worked around these issues, but I spent a fair amount of time recently figuring out how to address all of these issues more directly.  With some additional configuration steps and a little bit of editor scripting, all of these issues can be addressed quite nicely. Continue reading “Extending Unity for Git (and maybe other source control?)”

Optimizing Networked Strings as a Prebuild Step

Lately, I’ve been working on a fast-paced VR multiplayer shooter game in Unity with some classmates. Since we’ve had negative experiences with UNET in the past and knew we would need to optimize the netcode pretty carefully if the project was to play well, we decided to build a custom networking layer on top of the fantastic Lidgren UDP library. Most of my time has gone into building the networking layer from the ground up (which has been a total blast). Continue reading “Optimizing Networked Strings as a Prebuild Step”

Wave simulation is hard, or why you should only compute what you have to

I almost called this “Why you should always remember that games are illusions”, but that sounded way too pretentious and, honestly, kind of sad.

This weekend I participated Global Game Jam 2017, wherein the theme was “Waves”.  Our game, Bermuda Dieangle, involved controlling the ocean surface to make boats crash into each other.  Initially, we planned on running a complex wave simulation to make the boat and water physics extremely realistic and even had code in place to approximate it (one iteration on the CPU running on a background thread which ended up being too slow, and another using the GPU to propagate the wave forces, which we never got to work quite right).  But in the end, given our relatively limited experience in physical simulation and extremely limited time, we decided to fake it.  Without going into too much detail, each wave was essentially represented by a sphere which grew until it had no more energy.  Each boat checked if it was within the sphere of influence of any of these waves and, if so, applied a force away from the wave’s origin.  Since there were so few waves per frame, this was really cheap to compute and easy to write.  For the visualization, we took this data and generated a heightmap of the ocean surface by rendering each wave to a render texture as the sine of (the distance from the center / the current radius of the wave) + time + a random value unique to each wave (specifically, we had an orthographic camera pointing down towards the fluid surface and drew a radial texture with extra math being done in the fragment shader for each wave).  This was also extremely cheap and ended up looking really, really good (especially for the low-poly aesthetic we were trying to achieve).  Not only that, we were able to implement it in a couple of hours, compared to the 10+ hours we spent trying to figure out how to run a more physically accurate fluid simulation which never actually worked.  Furthermore, it should theoretically run on mobile or the web, as opposed to either of the more complex solutions where that would have been totally impractical.  And since all of the computations are pretty simple, buoyancy becomes feasible without copying any data from the GPU to the CPU because you only have to sample at a few points per ship/per wave (even though our buoyancy was broken at the end, it should be pretty simple to get in and fix).

So in short, don’t bother computing anything that you don’t have to.  And for the love of all that is holy, if you’re a bunch of college kids with very little experience in wave dynamics and not enough time to do thorough research, don’t try to write a real-time wave simulation/propagation system in a hackathon where that’s only a small part of the project.  Just don’t.