:format(webp)/cdn0.vox-cdn.com/uploads/chorus_image/image/37551576/Leap_Motion_VR_Developer_Mount_on_HMD.0.0.jpg)
One of the best features of VR headsets also happens to be one of its greatest shortcomings. Headsets are designed so that you can't see anything else, making games incredibly immersive. Yet it also means that interacting with the real world requires removing it from your head. Not doing so brings the risk of knocking things over, including yourself. San Francisco-based Leap Motion has come up with a solution: you attach its $79.99 infrared camera sensor to the front of your headset, and use software to display live video of the real world right in the mask. Even better, you can do it without fully leaving whatever game or virtual experience you're in, opening the door to new augmented realities.
VR's greatest feature is also a safety problem
Leap has been toying with this idea internally for a while now, but the company wants to make it an officially supported feature, and (more importantly) the future of its business. Today it's putting its hat in the ring to be the go-to camera for VR headsets. It's doing that in two ways: a standard mount that will let you stick one of Leap Motion's controllers to the front of any VR headset, and a next-generation color camera codenamed Dragonfly that it wants to be built into the front of all future VR headsets. Developers can buy the mount starting today, but Dragonfly is still a work in progress and will require buy-in from companies like Oculus, Sony, and others.
In a brief demo this week, we used a prototype version of Dragonfly mounted to the front of the latest development version of the Oculus Rift. Using software, Leap can jump between the video from its camera to software being displayed through the Rift's screen. It can also blend those two things together into one, so you can be in two places at once. In practice, this meant that I could hold out my arms and actually see them through the camera, as Leap's software overlayed what it was picking up from my hand's skeletal and joint structure.
Leap's big pitch so far has been a decidedly more stationary experience. Its small sensor bar plugs into computers and picks up specific finger and hand movements. It's been compared to Microsoft's Kinect, though co-founder and CEO Michael Buckwald argues that Leap is more suited to mimic exactly what we do with our hands rather than pick up full-body movements and simple gestures. Think less Fruit Ninja, and more Jenga. Developers have built a little more than 200 games and app experiences that use this, anything from floating around Google Earth to games that involve manipulating objects as you would with your hands.
But there's little arguing that the main product has not set the world on fire. It's only appeared on around a dozen machines made by HP, as well as a keyboard. Leap also laid off around 10 percent of its staff back in May, just months after launch, due to lower than expected sales. The company was hoping to sell 5 million units, but that number was reportedly closer to 500,000. Buckwald declined to say just how many the company's sold since launching a year ago, but was keen to point to the more than 100,000 developers in about 100 countries that have been building things.
"All of our focus has been on the PC."
The reason for Leap's shift to VR is simple. "All of our focus has been on the PC," Buckwald says. The only problem is that few PC-makers wanted to build Leap's finger- and hand-tracking technology into their hardware. By comparison, the VR market is still very much in its beginnings, and Buckwald believes that the companies making these headsets will see the Leap as something that can solve not only the vision problem, but also serve as a reliable sensor to track what people are doing with their hands.
"We think that unsolved problems with VR is input, specifically 3D input that is easy and accessible, but also powerful," Buckwald says. But Leap's sensor was not initially suited for VR headsets that need to capture your hand movements from the top down. In fact, the product was designed to capture whatever was above it. Leap had to reengineer its software to capture things that way. It's also had to open it up more to suit VR, giving developers access to the raw imaging data being picked up by the onboard sensors.
Leap Motion's new mount locks the sensor into place on VR headsets.
You can't fix everything with software though. Dragonfly, a sensor with two high-resolution, high-speed color cameras, is the result of that. Where Leap's current model can only see in infrared, the Dragonfly's color makes viewing the outside world a little closer to normal human vision. "It's something sort of essential to even be able to see at all," says Leap co-founder and chief technology officer David Holz. "If cameras are opaque, you're going to have these dual-purpose sensors that work as real eyes."
But helping VR users view the outside world in a way that isn't incredibly jarring is a complex endeavor, and one that isn't finished yet. Dragonfly itself is still in a prototype phase, as well as the software, which requires calibrating its cameras for each specific VR headset. The Dragonfly has a much wider field of view than the Rift, and it needs cropping and adjustment to work with the device's magnifying lenses. The scale and field of view also need to be accurate enough that you can interact with things without feeling disconnected from your body. Leap aims to take care of all these things for each individual VR mask, so that there's as little fiddling as possible.
People were asking for something other than duct tape
Fiddling is also one of the main reasons the company has come up with its own mount, which it's selling for $19.99. Developers that were strapping the Leap to the front of the Oculus Rift were looking for official guidance, and didn't want to start designing things that were only going to work for some users. "There are developers who asked how we mounted it on there, and we might say something like Duct tape," Holz says. "But they'd be like, ‘No, really.'"
As for the bigger plan of getting VR companies to build Leap's technology in, that's a bit more complicated than some adhesive. It requires that companies like Oculus, Sony, and soon Samsung, buy into Leap's idea. "We're talking to everyone in the space," Holz says. "But it's very easy to imagine this as a core part of the most of those existing products."