Kinect Mocap

When I first saw a Kinect, I immediately thought about using it as a low cost motion capture device.  I’m sure every developer did.  I bought one as soon as I could and played around with it, using some open source stuff to get at the bits.

It was quite awful, the jitter was measured in feet rather than inches, and it would often completely flip out, bones flying everywhere.

So a week or so ago I was attempting to animate a walk cycle and decided to give it another go with the official Microsoft SDK stuff.  Recording data was a breeze, and I had files saving full of data in five minutes.  I could immediately see that the results were much improved now that they have some built in smoothing and jitter correction.

The method I’m using is to take the raw captured SkeletonStream frames and apply a map to them to adjust each bone to fit the skeleton in use.  Here’s the GUI I’m using and the values:

The spine needed a little tweak, and the hip stuff is a coordinate system change + the high angle I am using.  The hip is the root.  The legs and hand are likely just a twist in the bones.  I am using mixamo’s awesome autorigging lately, and I’m really impressed with it.

Here’s the dirty old ladder I’m using to hold the camera:

I leveled it with some magazines:

I experimented a bit with different clothing.  For arms and legs, tight black stuff seems to be the way to go.  For hands and feet, I am not really sure.  I tried black gloves and socks, and white, and I think white worked better.  For all I know the clothing color might not even matter.

The tool itself is a collada reader / converter.  I wrote it a couple years ago to handle static and animated meshes for my XNA stuff.  Getting the mixamo and Kinect stuff working with it revealed a ton of flaws, so it took me a few days to beat it into shape.  I really only supported the default max collada exporter, which I later discovered doesn’t follow the spec very well.  Now I also support the … feeling… I think that is the name, that exporter.  Here’s how it looks set up to start capturing:

So how well does it work?  Well it is very situational.  The skeleton stuff can only figure things out if you are standing straight in front of the camera in a very small spot.  Vary even a meter and it starts to get really confused.  Turn around or assume a strange pose and that will confuse it as well.  Also having bits occluded will confuse it.  Here’s a really simple test I recorded to help align the arms: 

For my own needs, I’m thinking it will come in handy for situational stuff in future games.  Where I need the most help though is in walk/run cycles.  I’m no animator, and I find both of these extremely difficult and time consuming, so I was really hoping it would help with that.

I’m hoping to find someone local with a treadmill to try capturing runs and walks in place.  If anyone has any other ideas I’d love to hear them.

Sadly working on this delayed my one-game-a-month challenge stuff.  I’ve missed January and February now, mainly because I decided to do a platformer / sidescroller as my first game.  This turned out to be a mistake as I don’t really like that genre of games (aside from Braid) so I couldn’t really get into working on it.  I did make a neat looking glass lobby with an elevator though:

I abandoned that and began working on a top down perspective shoot & loot game.  At the same time during all of this I’ve been learning web stuff, entity framework and webforms and stuff.  It is BIZZARO LAND!  Web people are completely insane, but I have been having a lot of fun learning it.  More on that another day!