A technical note on Vogels! and a not so technical note on Komodo Games and its success
Before I start with the technicalities, let me give first a little background info on Komodo Games and Vogels!
During the second semester of 2010, I was involved in a unique game experimentation-project. Our team consisted of seven members, four from the Utrecht School of the Arts (HKU) together with me and Jens from the Game and Media Technology program at Utrecht University. Our project needed a name, so during our first brainstorm session, we came up with the name Komodo Games. The diversity of our team was evident from the roles each member had. Our group consisted of a Game Designer, Technical Game designer, Sound designer, 2D artist, 3D character designer, 3D environmental artist and me as the developer. Or more specifically:
- Adriaan de Jongh — Game Designer
- Jens van de Water — Technical Game Designer
- Eri Shiroyama — Sound Designer
- Sandra da Cruz Martins — 2D Artist
- Tim Remmers — 3D Character Designer
- Ronald Houtermans — 3D Environment Artist
- Francis Laclé — Developer
All in all, a diverse group with specific roles that needed to complete a game project in 22 weeks (more or less). So, what is Vogels! and how did we come up with the idea in the first place?
Vogels! (or Birds! in Dutch) is a serious game designed to support rehabilitation for patients suffering from Hemiparesis, a condition where one side of the body suffers from mobility impairment. The company FOCAL Meditech BV based in Tilburg, The Netherlands was the company that supplied the hardware for our serious game. They specialize in making custom wheelchairs for such patients, and over the last few years had been doing some research into games as a sort of add-on for the rehabilitation process. One of the most important types of rehabilitation is improving the mobility of one’s arm. If you think about it, you use your arm daily for almost everything. Having lost control of your arm would, in fact, be worse than losing control of your legs because, for instance, you couldn’t propel yourself when sitting in a wheelchair, or you couldn’t easily feed yourself. For this specific type of rehabilitation, they created a mechanical arm-support prototype with counterweights to allow a patient to focus just on the movements and less on the weight of their arm. They already had a 2D game prototype and wanted us to extend the research and come up with a prototype of a game in 3D. Why 3D? Well, 3D meant using their device for tracking movements in all three axes, and so patients would be able to train their arm muscles in six degrees of freedom. During our research phase, a few of us paid a visit to some clinics to get an idea for possible themes and colors that were relevant to our target audience. We combined that information with information we got from physiotherapists on the range of movements that these patients would execute during a standard therapy session. This turned out to be more than useful because the combined information had a direct consequence on the game mechanic that we would eventually adopt. Having iterated a few times over the results of our research, we came up with the idea of having a bird constantly flying around with the camera placed right behind, constantly targeting our protagonist. This allowed us to utilize all three axes and helped us to intuitively translate the mechanical movements of the hardware to the in-game movements of our bird and thereby for our patients. This was a critical criterion for our project because some patients had never played a computer game or never even sat behind a computer before.
It's all about motion nowadays
Because this game relied heavily on its unique control system, I was put in charge of programming the capturing of the movement data and sending that to our game-engine in real-time. Not a very straightforward task. One fellow team-member (Jens, our awesome technical game designer) was in charge of making sure that what you did with the controller was what you saw in the game. That meant computing the possible range of movements, setting up a calibration scheme and implementing linear interpolation to smooth the movements of the bird without causing noticeable motion lag. It also meant designing visual guides (in the form of wind particles) and turning the bird sideways when it changed directions, thus simulating a banking-rotation (see video here below).
At the very beginning, we were using the Wii Remote to capture the data. However, we had numerous difficulties with Unity, our game-engine. Because Unity used at the time an outdated version of Mono, the Wii Remote libraries were incompatible or weren’t working 100%. Having discussed the issue with FOCAL Meditech it was suggested to find another way to capture the data, not just for technical reasons but also business-wise. Having a custom method for motion tracking is in many ways better than using components from an off-the-shelf game controller, even if it’s just a prototype.
Arno Kamphuis, our supervisor at the university, proposed capturing the motion data using OpenCV. I ended up using a wrapper for OpenCV in C# called EmguCV for the following reasons. After doing some research online, it turned out that EmguCV was one of the best recommended wrappers that offered a high-level programming layer. This meant it would cost less time to create that same functionality in something like C++, which is the default language of OpenCV. Coming mostly from a Java/PHP background, this was the first time that I used C# in a real project, so I also learned a thing or two. The third reason for a C# wrapper was the compatibility it provided with our Unity game engine. You can import C# scripts and compile them directly from within the engine (through Mono). Later it turned out that this was not the way to go because the Mono library was outdated, and I wasted a lot of time trying to see if it was still possible to combine it with EmguCV. Overall, it turned out for the better for business to separate the two layers because then you can make the input manager independent of the game engine.
So without the issues that we were having with the Wii Remote and finally coming up with the decision to use a separate piece of software to track the input, I will now explain how the system works without giving too much technical information. The arm has four LEDs positioned at equal distance from each other. We capture a filtered image through OpenCV with a camera that is placed on the top (pictured above). This gave us four feature points. With these feature points, we can make a square, which will then give us the perimeter. This is useful if we want to track the vertical height. In other words, the greater the perimeter, the smaller the distance between the arm and the camera. Rotation for a certain degree can also be obtained, but more time was needed to implement it, and it doesn’t work for fast-moving motions. This could be technically solved with some hardware adjustments, but in the end we just didn’t have enough time to achieve the same robustness that we got from the tracking algorithm, so we decided to drop the feature at the last minute. On a positive note, one feature that did come in handy was point compensation. It would often happen that a patient occludes one LED with their arm. In this case, I needed to check which point was missing and virtually add that point to the tracking process. But because the shape is a standard square, there was no way to track movement when two points or more were occluded. For tracking, you need at least a triangle shape (or three points). Nevertheless, it lowered the probability that the points wouldn’t get tracked.
The final idea was to have a separate running application and send the data through a network protocol (pictured above). This brings off course some performance issues, so we had to carry out some test scenarios to improve the performance quite a bit, both for the motion tracking software as in Unity. Besides performance, there were many things that could have been done nicer on the technical side. Personally, the most important programming lesson that I have learned is to not spend too much time on making incompatible software compatible, even if you are a stubborn perfectionist.
The Dutch Game Awards 2010
The awards were held this year for the Game In The City event in Amersfoort, The Netherlands, and I have to say what a pleasant event it was. Vogels! got nominated for three categories, Special Award, Best Student’s Game, Best Serious Game. In the end, we won the Special Award! No one on the team was expecting it, I think. The other nominees delivered strong products, so the competition was fierce (see video here below).
We also managed to get nominated (on the same day) for the Steve P. Wozniak (Wozzie) Award, which was really unexpected. Combing these nominations with the Diamond Trophy Award from last summer, I can say that we can be proud as a team. What made Vogels! attractive in my opinion is its unique input controller and what it means to use video games for good. The field of serious games is expanding fast and entering unknown territories every year. I hope that FOCAL Meditech releases a commercial product in the near future, and I’m expecting a growth in serious games thanks to the advances and innovations in computer vision, propelled by the Nintendo Wii revolution of the past few years.
Here we see Tōru Iwatani (creator of Pac-Man) testing out Vogels! @NLGD 2010 :)
Originally published at https://www.visualacuity.nl on December 3, 2010.
