FlightGear Demo

Ah, the miracle of simulated flight. We take a stab at creating 3D gesture controls for piloting an aircraft in FlightGear, a free, open source flight simulator.

Here's a video detailing the different gestures and the operations that they map to. Disclaimer: I'm not a pilot, or even a seasoned simulated pilot, so pardon my reckless maneuvers.

Let me give a more detailed explanation of the gestures.

Startup controls
We mapped the operations to each hand, based on how one might operate the controls in a real cockpit. The throttle and the parking brake can only be accessed by pinching with the right hand. The physical interaction space in front of the monitor was divided such that pinching in the rightmost area of the interaction space could trigger only the throttle. Moving the hand forward and back mimics the physical act of pushing a throttle forward and back. Pinching in the center of the interaction space accesses the parking brake. We use a rotation of the hand to reflect the motion of the parking brake being released or set. Finally, the ignition is located on the left side of the flight controls, so we use a left hand pinch to access the ignition. Rotating clockwise closely mimics the act of turning a key.

Flight controls
Flight controls only become engaged when both hands perform a pointing gestures. I admit that a real pilot probably doesn't point his or her fingers while piloting a plane, but this is a demonstration of the different hand poses we are working on exposing to the developer. Robust recognition of different hand poses would allow the developer to use hand pose to set the mode of the application. The two hands control three things: ailerons, elevators, and rudders, which control the roll, pitch, and yaw of the aircraft respectively.

Ailerons:

Elevators:

Rudders:

In summary the gestures are as follows:

• Throttle - A pinch with the right hand, pushing forward and back, in the right side of the interaction space
• Parking brake - A pinch with the right hand, rotating clockwise to releast the break and counter-clockwise to set the break, in the center of the interaction space
• Ignition - A pinch with the left hand, rotating clockwise
• Ailerons - Pointing with both hands, moving the hands like turning a vertical steering wheel
• Elevators - Pointing with both hands, rotating both hands upwards or downwards
• Rudders - Pointing with both hands, moving the hands like turning a horizontal steering wheel

Kenrick

Recipe Demo

Hi!

I thought I'd share a little demo I put together using our hand tracking system. One of the advantages of using a touchless interface is it keeps your hands clean from dirty equipment and vice versa. For example, a surgeon who has scrubbed in does not want to handle unsterile equipment such as a computer. Conversely the surgeon does not want to contaminate equipment with the byproducts of surgery. Rather than constantly breaking scrubs or dictating instructions to another person, the surgeon could manipulate images and patient data through a touchless interface. Or imagine someone in waste management who needs to operate a computer or someone in the food industry using a shared computer touched by countless individuals. A touchless interface would keep the equipment and the user's hands clean.

Okay, so maybe you're not a surgeon and you don't work in the dirty jobs industry. But, have you ever cooked while following a recipe on a computer? You probably didn't want to get your food-covered hands all over your nice devices. I've cobbled together a simple demo that allows a budding pastry chef to mark off the steps in a recipe for chocolate chip cookies.

The gestures are simple. The Swipe and Lock gesture crosses off items on the list - with the right hand, the user can perform a pinch in the air and drag to the right to begin crossing off a step in the recipe. Then, rotating the hand clockwise roughly 90 degrees confirms the cross-off. A pinch and drag to the left undos the cross-off. A pinch and drag up or down traverses the steps. The user can also view the ingredients list by using the left hand to perform a double-pinch (like a double-click) and dragging to the right to reveal the second panel.

Check it out:

Following a recipe is a cute example, but there are other applications for checklists such as in a laboratory setting where a scientist might want to cross off steps in a procedure that requires aseptic techniques. Or in the operating room, the doctor might use a checklist as a cognitive aid while performing a procedure.

Kenrick