Tuesday, October 2, 2012

Hello from 3Gear Systems

Hello from 3Gear Systems!

We're a three-person team based out of San Francisco trying to fundamentally change the way people interact with computers. We're excited to kick things off on our blog by announcing the release of a software development kit (SDK) for adding gestures to your applications.

The story so far

It's easy to forget that the mouse is over 40 years old. (Source: Wikipedia / SRI International)

It's easy to forget that the mouse is over 40 years old. While today's mice are smoother and have more buttons, they haven't changed all that much. The biggest innovation in UI since the mouse has been the touchscreen, which, like the mouse, treats your hand as if it's one big pointing finger (or at most two fingers sliding around pictures under glass).

But your hands can do so much more than point at things! They can grab things, turn things over, assemble things, animate things, etc.

At 3Gear, we're creating technology that uses your entire hand (fingers, thumbs, wrists and all) for user interaction. This is especially useful when you're doing something 3D, say assembling 3D parts in Computer Aided Design (CAD), flying through a medical 3D MRI scan, or playing 3D games. With the rise of 3D printing and the Maker community, we're especially interested in making it easier to create in 3D.

Making the Kinect "finger-precise"

We're using 3D camera hardware (e.g., the Microsoft Kinect) to make this possible. However, existing Kinect software only work on large, full-body motions. We've developed software that creates a finger-precise representation of what your hands are doing, capturing tiny motions of your index finger and subtle movements of the wrist. This means your applications can use small, comfortable gestures such as pinching and pointing rather than sweeping arm motions.

To make this work, we had to develop new computer graphics algorithms for reconstructing the precise pose of the user's hands from 3D cameras. A key component of the algorithm is to use a database of pre-computed 3D images corresponding to each possible hand configuration in the workspace. The 3D image database is efficiently sampled and indexed to enable extremely fast searches. At run-time, the images from the 3D cameras are used to "look up" the pose of the hand using the database. This way, the user's hand pose can be determined within milliseconds — fast enough for interactive applications and a short enough time to avoid the effects of "lag" or high latency.

Gestures you can use all day

Because the cameras are mounted above the desk, it is still possible to use our system even when the hands are very close to the desk surface.

Before founding 3Gear, Rob spent a lot of his PhD working on tracking hands with a color patchwork glove. He eventually abandoned the idea because it was so hard to get people to put on gloves. Above all, our input system is designed to be practical and comfortable. We're trying to fit into your workflow rather than interrupt it. We've been careful to design an input device that you can use all day. For instance, we mount the cameras above the desk so that the hands can be tracked well even a couple centimeters above the keyboard with the forearms resting on the desk, avoiding the so-called "gorilla arm" problem. You don't have to wave your arm around much to interact with the system; it can pick up motions on the order of millimeters.

Try our software development kit (SDK)!

Our input system uses two Kinect cameras and an aluminum frame for mounting the cameras.

Users can grab virtual objects and move them around in 3D with their hands.

Today, we're releasing a "public beta" version of a software development kit (SDK) that allows you to quickly incorporate 3Gear's technology into your applications or invent new uses of gestural user interfaces.

Here's what our software can do:

  • Intuitive 3D manipulation. Our 3D input technology provides 1-to-1 3D control of virtual objects. Users can grab objects and move them around in 3D with their hands.
  • Touchless (aseptic) control. Whether the user is a surgeon in an operating room or a chemist in a pharmaceutical lab, sometimes it's just too inconvenient to touch a computer. Our technology offers precise control of computer systems for dirty jobs.
  • Runs on commodity hardware. Our input system currently uses two Kinect cameras and an aluminum frame for mounting the cameras. All of the components are available off-the-shelf right now.
And here are the limitations:
  • Our software requires each first-time user to go through a short (five-minute) calibration step. A user only ever has to do this once ever.
  • It's good at recognizing the set of useful gestures covered in the calibration, but it can't track arbitrary hand gestures quite yet.

We're working hard to relax these limitations. We're also actively trying out new camera tech and frame designs — we know it looks a little clunky right now.

Our software is free for both non-commercial and commercial applications up until the end of the beta period (November 30th, 2012). After the beta period, we will continue to offer a free version of the software for researchers, hobbyists, and small commercial entities (i.e., annual turnover of US$100,000 or less).


About us

Okay, so since this is our first post, here's a little more about us. We're just three people right now, but we've all been thinking about computer graphics and HCI for a while:

Rob Wang wrote his doctoral dissertation at MIT on tracking colorful things using computer vision. Most notably, he created a color patchwork glove and a set of algorithms for tracking a user's hands in real-time with a webcam.

Chris Twigg figured out how to step back in time while getting his PhD at Carnegie Mellon University. Before founding 3Gear Systems, Chris pushed the envelope of digital visual effects at Industrial Light and Magic R&D.

Kenrick Kin got his Ph.D. from Cal but spent much of his time in grad school at Pixar Animation Studios inventing ways to use multitouch screens to build rich 3D environments for computer-animated films.

We're supported by K9 Ventures, Uj Ventures and a research grant from the National Science Foundation

“Kinect” is a registered trademark of Microsoft.

15 comments:

  1. Hi young Gents / n future Ladies, Thomas here, of Vancouver Canada. i am an old Techno fart, of System integration skills, Control Consoles, Radio system control, remote access control and monitoring , control rooms, , i see great promise in your efforts, Making the hands to by remote control, knob tweeking , all so possible by your design. i am going to look close, shift some of my time, to look at, and perhaps incorporate into my designs. Cheers from Thomas in Vancouver Canada

    ReplyDelete
  2. Rolf Thomas Owan Gx 0 minutes ago
    Hold on Boys and Girls, besides an interface for "Gamers" this is an industrial control interface. Meaning you can control instrumentation by remote control, from close by, / next room, to miles away. state's away, even other countries. there has been various forms of cruder interface by approximation for years. this is just a refinement, by some very bright and hardworking young men. i give thumbs up, and will try to make some free time to explore this. " i actually think , this type of control ' has been long overdue. i will probably get the kit, and try a software interface for some of my remote control applications, , i like the idea of non user specific control capability, / i have a few good ideas to mature.

    ReplyDelete
  3. George Gallo ... This is mind boggling but then I realize that computer games are doing this already.
    18 minutes ago

    Yes George, it is, as an example, say good sign language people come into a restaurant, and want to order, well they do it with sign language, and the order taker, has it down Pat, exactly and the table readout anouncer cube gives the order in english, / other languages, and masculine and femine voice. . so cool, i think it is going to happen

    ReplyDelete
  4. Add a 3d capable screen linked with 3d glasses, and add Autodesk functionality and I will draw all houses with this

    ReplyDelete
  5. Wow... this could make Blender something that people can easily use.

    ReplyDelete
  6. It's truly nice software and fantastic and useful blog thanks for publishing useful and information great to read it.
    Software Company Lucknow

    ReplyDelete
  7. Hi !
    Looking forward for C# support! Amazing Job :)
    Regards from Poland :)

    ReplyDelete
  8. Very Nice and informative blog post. this blog have really great information and your software development kit description is good. It touch less (aseptic) control is impressive. Software Development Lucknow

    ReplyDelete
  9. Thank you for sharing this informative blog post. Software development kits always helpful because lot of tools meet at one place so tool kits are so helpful. Web Development Company India

    ReplyDelete
  10. Kits always be helpful in software development because in kit there are many required software exist, which is easy to use. Software Company in Lucknow

    ReplyDelete
  11. There are lots of great choices now when it comes on home security system. You can ask for a professional guide when selecting one that would fit to your needs.
    security consulting.

    ReplyDelete
  12. Its really great blog to get the information regarding 3 gear system in software development.Its really interesting.
    Web Design Companies | Web Development Company

    ReplyDelete
  13. This is very good common sense blog that is the main thing a lot of interesting and appreciable to all. You are post some really good points 3 gear system in software development. In the present time Information technology has one of the most promising industries across the globe. Great experienced to learn this post.
    Software Company Lucknow

    ReplyDelete
  14. We are a leading software development company delhi,India which works as per the client requirements and give provide software.

    ReplyDelete