Hackers worldwide Kinect with their creative sides

Japanese developers are using a Microsoft Xbox 360 Kinect hack as their playground

by Verena Dauerer

When Kinect, Microsoft’s latest add-on for the Xbox 360 game console, was released worldwide in November 2010, it was the beginning of a success story, but not in the way Microsoft may have been expecting.

The peripheral, which uses a camera and motion-capture software to allow for gaming without the use of an actual controller, has been like a flame to a moth for hackers. Consequently it wasn’t long before the potential of the system beyond gaming was being explored.

To the joy of the independant programmer community worldwide, only a week after the Kinect’s release, a hacker named Hector Martin developed an open-source driver that gets hold of the camera data so that anybody can experiment with the camera freely. Within days of the hack, numerous sites popped up devoting themselves entirely to the cause, such as kinecthacks.net.

Since then, programmers all over the world have enthusiastically embraced the device as a virtual playground. And despite the fact that the Japanese underground hacking community often shies away from sharing their projects with the outside world, many of the most interesting Kinect hacks are coming from Japan.

Takayuki Fukatsu from Yokosuka, Kanagawa Prefecture, has received worldwide recognition for the iPhone apps he has developed under his Art & Mobile brand, such as TiltShift Generator and QuadCamera. But recently he has been attracted by the novelty of hacking the Kinect. “It’s like a snowfield where no one has been,” he said via e-mail, “making a footprint on it is really fun, like child’s play.”

In his hack, Fukatsu came up with a camouflage-like experiment that makes him invisible on a realtime camera feed; floating like a ghost through a room with his silhouette only faintly perceivable when he moves. To achieve the effect, he configured the Kinect’s camera with openFrameworks, (an open-source toolkit to enhance the programming language C++), to generate the optical illusion of him being invisible on the camera’s live feed. This is possible because the Kinect takes data from a screenshot of the previously empty room. When Fukatsu enters the video area, the software detects his shape and maps the background image over him — making him invisible, and looking a lot like the cloaking camouflage used by the alien in the “Predator”movies!

A Kinect hack, however, can be practically anything, limited only by what the device can do and the imagination of the hacker.

Tomoto Shimizu Washio works at Hitachi Data Systems in San Francisco. When hacks began emerging, he too wanted to participate with something special. So, with the passion of a true fan, he set out to turn himself into Ultra Seven, a masked superhero TV character famous since the 1960s. In the video Washio released on YouTube he can be seen firing light beams and using his silver crest as a throwing weapon, before flying up, up and away.

“I felt as though I had been left behind somehow,” Washio explains, “and I wanted to stand on the same ground with others developers, be it only for a moment.” He succeeded: More than 300,000 viewers have watched the clips on YouTube so far. And he has released its open-source code for others to play with.

“When a new technology comes up, unless it is related to your regular business, it is usually difficult to catch up and do something with it in a timely manner. It may be too late by the time you notice,” he says.

To stay up to date, Washio taught himself the basics of C++ and a cross-platform Application Programming Interface (API) called Open Graphics Library (openGL).

First Washio generated a virtual 3-D space with animated objects, such as the beams. He then added the images taken with the Kinect camera, letting them overlap the virtual space and be combined on screen.

His inspiration for the project came from works by other Japanese programmers on the kinecthack.net web site. Though he isn’t connected to any of them, Washio says he thinks their projects are remarkable. And yet the language barrier seems to stop them getting more recognition. “I sometimes feel Japan suffers a disadvantage,” he says from his perspective living in San Francisco. “Its potential technical excellence is not as visible to the world as it should be.”

Fukatsu also blames the cultural divide. “We don’t join the English programmers’ community that much,” he says. But there is also another reason: Japan’s market size.

“The (domestic) market is big enough to become successful in,” explains Fukatsu. “Most developers and companies only develop for (the local market) because it is easy and low risk. It makes people a little defensive, and conservative.” He observes that many Japanese developers feel that making something for the overseas market is risky since they can’t join the English-speaking community.

Fukatsu’s main communication tool with other programmers in Japan is Twitter, and with 14,500 followers he has quite an audience. However, he thinks there should be more face-to-face gatherings.

This is an opinion shared by a fellow developer, Atsushi Tadokoro, from Setagaya, Tokyo. Tadokoro says, “It feels like there are a lot of talented programmers, but they don’t know about each other.” Despite the many gatherings for Adobe Flash or HTML developers, there are none for programming languages such as Processing or Cinder. An exception is the Tokyo Max User Group ( tokyomax.jp ) that has a monthly event for their live video performances. At the moment Tadokoro is planning to set up a Tokyo programmer community focusing on the openFrameworks user group.

In the meantime, Tadokoro has created quite a clever Kinect hack himself using openFrameworks and an extension called ofxKinect for the motion tracking. Inspired by the sci-fi thriller “Minority Report,” he came up with the idea for an interface to play music — generated only by hand gestures that draw lines in the air from various angles. He achieved this marvelous effect by programming the camera to measure the distance between it and the user and then having the software create rectangles in the space. The rectangles act as notes in a sequence as the user’s hands pass through them. Tadokoro explains, “The notes can have different lengths, and as the music loops around you can play it like a musical instrument. Also the tone and position of the note changes depending on where the rectangle is placed.”

All this has been achieved in the few months since the original Kinect hack was released. It will be fascinating to keep an eye on what programmers will come up with in the future — and these Japanese hackers have proved that language need not be a barrier.

To view the works mentioned above see: Takayuki Fukatsu: www.youtube.com/user/TakayukiFukatsu Tomoto Shimizu Washio: www.youtube.com/user/hogehoge335 Atsushi Tadokoro: vimeo.com/17240146