There is a fundamental problem when creating new hardware: you need software using it before anyone is willing to buy it. The problem with getting software written for new hardware, is that no one wants to put in the time to build applications using hardware that no one has bought yet. This chicken and egg problem has killed lots of cool hardware, from anyone who hasn't had the skills to develop their own killer apps, or the clout to convince the world that every one will buy their hardware. My first week of work on the challenge has been dedicated to trying to solve this problem.
The plan is to write a platform layer like GLUT and SDL, but with a twist. Instead of giving the application access to inputs like mouse and keyboard, I describe generic inputs like pointers, axis and events. That way an application that supports a pointer, works just as well if the input comes from a mouse, a touch screen, a Waccom, WiiMote or any other hardware that you can use to point with. Obviously they have differences you can query; A mouse may have multiple buttons, while a touch screen only has one "button". (You cant "right click" on a touch screen, well yet. If you build that hardware my API will support it).
Most platform libraries come as a DLL, and if we make this a open source DLL anyone could add what ever hardware support they wanted to it and any application using the DLL would be exposed to the new hardware. Great! Except if we wrote a DLL that supported every known hardware, it would obviously become huge and create a dependency hell, and what if someone created a version of the DLL that supported my sound system and someone else wrote a different version of the DLL that supported my pedals, how would I be able to play my racing game and hear the engine scream when I pushed the pedal?
So I decided we need a different approach, Lets make the library lean and mean instead, but lets give it a plugin interface so that you can write modules for it that add new functionality. Each module is independent and can add as much or as little functionality as you want. The library itself is only a few files large so you can even just drop them in to your project and make a self contained executable that has no dependencies to any DLL. Nice and tidy!
This week I set out to write this library dubbed "Betray" and I knew I was in for a bit of a Indirection hell but problems didn't arise where I thought they would.
The first objective was to do a bit of house keeping and create a utility library to handle some platform specific things, that aren't related to windows, drawing or inputs. In about a day I wrote the sub library "imagine" to handle the following:
-Directory management (Listing volumes and directories, had to do some
work to unify unix and windows here)
-My Application settings API (previously found in "Seduce")
-Dynamic loading of Libraries and sharing of function pointers.
(Needed for the plugin system)
-Threads and Mutexes. (Previously in my old platform layer)
-Execution. (Previously in my old platform layer)
Then I went on to implementing the basic out of the box functionality of the betray library: (Much of this was code taken form older projects)
-Opening a window with OpenGL/OpenGL ES Context (With FSAA)
-Mouse / keyboard
-Opening file requesters.
-Quad buffer stereoscopic. (It should work but i don't have a display
to test it on :-()
-Multi-touch (will still run on pre 7 Windows)
See the API HERE
This was quick and easy and I followed it by building a brand new plug-in API. It too went fairly pain less, although the constant passing around of function pointers got mind numbing after a while. Once done the new plug-in API supported:
Allocation and setting of:
-Buttons with labels and key-codes.
-The ability to hook in to the main loop.
-The ability to listen to events from windows event pump
-A sound API (Few features are still missing)
-A settings API, so that plugins can communicate their settings to a
See the API HERE
I started out writing some test plugins for some hardware I found in my apartment like a Microsoft 360 controller. It worked brilliantly once I figured out what the DLL needed was really called (Not what MSDN says). Then I went on to write a plugin for TrackIR and that went reasonably well too.
Then I had this idea that turned in to a Rabbit hole: What if the Betray API could trick the application in to drawing in to a texture instead of the the screen? Then (potentialy) a plugin could manipulate the screen output before its drawn to screen. You could do things like Color correction plugins (You could play Diablo as gloomy as you want!), plugins that could save out massive screen shots, and if you let the plugins draw more then once you could even support anaglyph 3D, and mult-iscreen CAVE environments!
This was just too cool to not do, so I wrote all the code I though I needed to do this. Then I ran the code... Well it did nothing I thought it would. The problem is that the application has a bunch of OpenGL state and as soon as the plugin tries to access any OpenGL functionality it will need to set its own state, and thats a problem because it, A doesn't know what state OpenGL is in, and B upsets the applications state. I briefly considered trying to read out the current state so that plugins could put it back once it was done with it, but that would be a huge amount of work and wont be forward compatible as newer versions of OpenGL adds more state. The solution will have to be to use 2 OpenGL contexts and its starting to get complex so I will need to do way more work on this.
Finally I came to the big price: The depth seeing camera intel sent me! I'm not at all convinced that depth seeing cameras are very good as interfaces but there is one particular feature Ive been looking for and that is the ability to get the vantage point of the user to the screen. A depth seeing camera should be able to very accurately compute the users head position in front of the computer.
Initially I had some problems just from the fact that the API is C++, and I am a pure C programmer but my good friend Pontus Nyman was nice enough to lend a hand and write a C wrapper for the functionality I needed. So one night we sat down to tie together his nice wrapper with my nice plugin API. Intel has provided us with the Perceptual computing API that contains face tracking so this should be easy, but when we started looking at the data coming out of it, it wasn't very good. It was jerky, imprecise and often didn't pickup a faces more then a few times a second. All the output turned out to be 2D and it leads me to believe it isn't using the depth camera to help it do better facial recognition (the depth camera output was less noisy then the color cameras). You do get access to the depth buffer, but its warped and you need to do a look-up in to a uv table to map it over to the color image, the problem is that you cant do it the other way around so its hard to look up the depth in the depth buffer of the facial detection running on the color buffer. We did some hacks to get something out of it, and for a brief moment here and there it was working, but not at all reliable enough.
I will give it a few more days, but right now its not looking very good. In theory I could write my own face detection code using the depth buffer alone that could be much better but that is a much larger project then i planed, for only a tangentially important feature. I want to begin work on my interface stuff this week, maybe its something I can look in to after GDC. This Week I intend to tie up all lose ends in the Betray platform, release it and move on to the new interface toolkit!
Edit: Intel confirms that the algorithm is not using the depth map for face recognition, but they also suspect I have faulty camera (I sent them images), so they are sending me a new one. The cameras are Pre-Production so this kind of thing is expected. Very nice to get such fast feedback!